In a
couple of
previous blogs, you learned about guaranteed delivery (GD) and how to use it with subscriptions in different scenarios. In this new series, I’ll talk about how to guarantee data delivery with SDK publishers, and give you an example using a Java SDK publisher.
Before I get to all that, let’s go through the prereqs for GD.
Prereqs for GD
The project that you are publishing to must:
- Have the consistent-recovery property set to true.
- Use a non-zero value for the auto-cp-trans property. This ensures that when the publisher issues a commit, the project processes the data fully before returning the commit call.
Note: For performance reasons, you should use a value less than 1000.
Whether publishing with SDKs or the Streaming Web Service (SWS), you must meet these requirements to guarantee data delivery.
Achieving GD with SDK publishers
To achieve GD with an SDK publisher, you must issue periodic commits. This ensures that data received by the server will be persisted in the log store. The number of commits that you issue depends on whether you are using synchronous or asynchronous publishing.
Sync mode
If you set sync mode to true (currently only possible if you choose the direct access publishing mode when creating the SDK publisher), frequent commits aren’t as necessary. This is because the SDK confirms whether data has been received by the server and queued in memory before sending the next batch. We call this an application level delivery guarantee. So, provided you issue a commit before project shutdown, all data will be processed. Nothing will fall through the cracks.
Async mode
The same, however, can’t be said for async (default) mode, because there is no application level delivery guarantee. Instead, the SDK continuously sends data without waiting for acknowledgment. Though this means better throughput, it also means that you can’t be sure whether the data has been received unless you issue a commit. So, if you issue more and more commits, you can be increasingly confident that the data is reaching its destination. Conversely, if you issue fewer commits, you’re more likely to need to roll data back.
Regardless of which mode you use, keep in mind that any data not committed before project shutdown will be lost. This is why periodic commits are so important. Like saving a document at certain checkpoints, periodic commits ensure that you don’t lose any progress if the program crashes.
Issuing a commit with SDK publishers
To achieve GD with SDK publishers, issue the following commit calls:
- C SDK: esp_publisher_commit(EspPublisher * publisher, EspError * error)
- .NET SDK: publisher.commit(NetEspError^ error)
- Java SDK: publisher.commit()
If you are issuing a commit after every publish (which you might choose to do in async mode to ensure all data is read), you should use an envelope or transaction of at least 1000 rows. Otherwise, the project performance will degrade because of frequent checkpoints. Also, when issuing commits, make sure that the publisher can resend the data if the commit fails. This is done with the ‘try’ and ‘catch’ tags, as you’ll see in the example code below. The ‘try’ tag tries to assemble and publish a message, and then tries to send a commit. The ‘catch’ tag tells you if there is an error, lets you know what failed, then disconnects, reconnects, and publishes again.
Example: Java SDK publisher
Here is an example of issuing a commit in async mode with a Java SDK publisher. This sample code is for illustration only; it does not comprise a complete, working example.
if (rowNum % batchSize == 0 || rowNum == totalRecs) {
try {
message.endBlock();
publisher.publish(message, true);
//Need to do a periodic commit to ensure that the project has persisted the data to disk
//Need to be able to resend the data if the commit fails
publisher.commit();
System.out.printIn("Successfully committed Row Num " + String.valueOf(rowNum) + ". ");
message.startEnvelope(batchSize);
} catch(Exception e) {
System.out.printIn("Failed to commit Row Num " + Sring.valueOf(rowNum) + ". " + e.getMessage());
//Retry connecting to the project every second until it connects.
In addition to SDK publishers, GD is also supported by the SWS. Keep your eyes peeled for another blog that covers this. It will include achieving GD with REST publish requests and WebSocket publishers.
For more information on SDK publishers, check out:
Java SDK: Publishing
C SDK: Publishing
.NET SDK: Publishing
