In a
previous blog, I talked about how to guarantee data delivery with SDK publishers. This time around, I’ll get into how to achieve GD with Streaming Web Service (SWS) publishers using REST and WebSocket connections. In each section, I’ll point you to a file in the streaming analytics installation where you can see an example in action. These example files are included in both the server and client packages of streaming analytics SP 02 revision 022 and newer.
Before reading on, remember that to achieve GD, the project you’re publishing to must:
- have the consistent-recovery property set to true, and
- use a non-zero value for the auto-cp-trans property.
SWS publishing
When publishing to a project with SWS, you can guarantee delivery by requesting SWS to issue a commit after publishing data. Support for issuing commits was added in:
- 2.0 SP 00 for REST publishing
- 2.0 SP 02 for WebSocket publishing
WebSocket publishing only requires periodic commits for GD to work. To guarantee in-order delivery with REST publishing, however, SWS must issue a commit with each publish request.
Publishing with REST requests
In 2.0 SP 00 and later versions, you can achieve GD by setting the
commit property to true in the publish request (the default value is false) each time you send a REST request to publish a message to a stream or window. By doing this, you’re telling SWS to issue a commit to the project after publishing rows. A commit must be issued for every REST publish request to ensure that data is published to the project in order.
For example, consider publish requests R1, R2, R3, and R4, where only R4 requests a commit. If the project goes down after R1 is published and comes back up before R4 is published, this results in out-of-order delivery where only R1 and R4 are published and committed. For in-order GD with REST publishing, you need to request SWS to issue a commit in each publish request and wait for the request to succeed before sending the next one.
To see an example of REST publishing with guaranteed delivery, check out the code example in
$STREAMING_HOME/examples/python/swsgd/rest_example.py.
Publishing with a WebSocket connection
As of 2.0 SP 02, you can guarantee data delivery when publishing with a WebSocket connection by connecting in commit mode (the default is simplex mode). Commit mode lets you:
- issue periodic commits by specifying the commit property as true in the publish message (the default value is false), and
- asynchronously receive response messages from SWS indicating whether publish and commit requests have succeeded or failed.
Unlike REST publishing, when publishing using a WebSocket connection you don’t need to issue a commit with each publish request or message. On a publish/commit failure or connection error while SWS is processing a publish message, subsequent messages are dropped until the connection is reset. The connection is reset when the client sends a “connection-reset” message to the SWS.
So, for the previous example with R1-R4 where only R4 requests a commit, if the project goes down after R1 is published, R2-R4 (and any subsequent messages) are dropped until the connection is reset. The client must send a “connection-reset” message and resend R1-R4. This ensures in-order delivery of data.
To see an example of WebSocket publishing with guaranteed delivery, check out the code example in
$STREAMING_HOME/examples/python/swsgd/websocket_example.py.
For more info about publishing with SWS, check out:
