In the first blog in this series, I talked about what guaranteed delivery is, how it works, and why you should probably use it for your streaming projects and adapters. If you missed it, you can read the introduction here.
Now, in the second part, I’ll be discussing how to use guaranteed delivery with subscriptions in three different scenarios: for bindings, for streaming output adapters, and for SDK subscribers.
If you are using a binding between a stream or window in two different projects, make sure you add the necessary guaranteed delivery options in the project configuration (CCR) file for your project, either through studio:
or through a text editor:
Since the windows used in this example are both input windows, <Output> must be set to true for the binding to work correctly – the source input window will not correctly transmit data to the input window in the other project otherwise.
<EnableGD> must be set to true, or Enable Guaranteed Delivery must be checked in studio for the binding to use guaranteed delivery. Give a unique name for the GD session (unique_session_1), and either set a batch size, or enable the GD cache if the source window is in GD mode with checkpoint.
The batch size will determine how many transactions are collected until the source window sends them to the target window, and the binding issues a GD commit. The GD cache will cache incoming transactions until the binding receives a checkpoint message from the source window – then it will send all transactions and issue a commit.
To use GD on an output adapter, the first step is to use an adapter that supports GD. You can check the adapters summary table in the Adapters Guide to see which adapters support guaranteed delivery, and which parameters need to be set for the adapter you want to use.
Each adapter will have parameters for enabling GD. These parameters can be set in studio by opening the properties dialog for that adapter. For example, the enableGDMode parameter must be set to true for the adapter to run in guaranteed delivery mode.
The enableGDcache parameter defaults to true, which causes the adapter to only send rows that can be recovered on restart by the smart data streaming server to its external service, and internally caches all others. But, using a GD cache with the adapter may cause an increase in latency, depending on how frequently the stream or window the adapter is attached to issues checkpoint messages.
The gdBatchSize determines how many message blocks it takes before the adapter issues a commit command to its service and a GD commit to its attached stream or window, and as a result, how soon the data rows it receives can be used by its service.
You can also add the corresponding CCL code to your project to set the same parameters, like in the example using the SAP HANA Output adapter below:
Make sure to set all necessary GD parameters for your adapter, and choose the right values for your project’s needs.
In this example, I’ll be using the .NET SDK, though the steps are very similar across all three SDKs. You can take a look at the methods here, and then take a look at the specifics for the C and Java SDKs in the SDK Guide.
1. Request a GD subscription by calling:
NetEspSubscriberOptions.set_gd_session(string session_name
and creating a NetEspSubscriber object.
2. Create and connect a NetEspPublisher object.
3. Retrieve the checkpoint sequence number for the last checkpointed data by calling:
NetEspSubscriberEvent.get_checkpoint_sequence_number(NetEspError^ error).
4. Tell the server that the subscriber has committed messages up to a given sequence number and no longer needs them by calling:
NetEspPublisher.commit_gd(String^ session_name, array<int32_t>^ stream_ids, array<int64_t>^ seq_nos, NetEspError^ error).
5. Cancel the GD session to free up resources once it is no longer needed by closing the subscriber or by calling:
NetEspProject.cancel_gd_subscriber_session(String^ gd_session, NetEspError^ error).
Take a look at the SDK Guide for a more complete example of using guaranteed delivery with SDK subscribers.
For more information on working with Guaranteed Delivery and Consistent Recovery:
Now, in the second part, I’ll be discussing how to use guaranteed delivery with subscriptions in three different scenarios: for bindings, for streaming output adapters, and for SDK subscribers.
How to subscribe with guaranteed delivery for project bindings
If you are using a binding between a stream or window in two different projects, make sure you add the necessary guaranteed delivery options in the project configuration (CCR) file for your project, either through studio:
or through a text editor:
<Bindings>
<Binding name="positions">
<Cluster>esp://localhost:9786</Cluster>
<Workspace>default</Workspace>
<Project>ccl_function_example</Project>
<BindingName>positions</BindingName>
<RemoteStream>TradeWindow</RemoteStream>
<Output>true</Output>
<EnableGD>true</EnableGD>
<GDName>unique_session_1</GDName>
<GDBatchSize>10</GDBatchSize>
<EnableGDCache>true</EnableGDCache>
</Binding>
</Bindings>Since the windows used in this example are both input windows, <Output> must be set to true for the binding to work correctly – the source input window will not correctly transmit data to the input window in the other project otherwise.
<EnableGD> must be set to true, or Enable Guaranteed Delivery must be checked in studio for the binding to use guaranteed delivery. Give a unique name for the GD session (unique_session_1), and either set a batch size, or enable the GD cache if the source window is in GD mode with checkpoint.
The batch size will determine how many transactions are collected until the source window sends them to the target window, and the binding issues a GD commit. The GD cache will cache incoming transactions until the binding receives a checkpoint message from the source window – then it will send all transactions and issue a commit.
How to subscribe with guaranteed delivery for output adapters
To use GD on an output adapter, the first step is to use an adapter that supports GD. You can check the adapters summary table in the Adapters Guide to see which adapters support guaranteed delivery, and which parameters need to be set for the adapter you want to use.
Each adapter will have parameters for enabling GD. These parameters can be set in studio by opening the properties dialog for that adapter. For example, the enableGDMode parameter must be set to true for the adapter to run in guaranteed delivery mode.
The enableGDcache parameter defaults to true, which causes the adapter to only send rows that can be recovered on restart by the smart data streaming server to its external service, and internally caches all others. But, using a GD cache with the adapter may cause an increase in latency, depending on how frequently the stream or window the adapter is attached to issues checkpoint messages.
The gdBatchSize determines how many message blocks it takes before the adapter issues a commit command to its service and a GD commit to its attached stream or window, and as a result, how soon the data rows it receives can be used by its service.
You can also add the corresponding CCL code to your project to set the same parameters, like in the example using the SAP HANA Output adapter below:
ATTACH OUTPUT ADAPTER saphana TYPE hana_out
TO ValueByBook
PROPERTIES
service = 'newservice2' ,
table = 'exampletable' ,
enableGDMode = TRUE ,
gdBatchSize = 15 ;Make sure to set all necessary GD parameters for your adapter, and choose the right values for your project’s needs.
How to subscribe with guaranteed delivery for SDK subscribers
In this example, I’ll be using the .NET SDK, though the steps are very similar across all three SDKs. You can take a look at the methods here, and then take a look at the specifics for the C and Java SDKs in the SDK Guide.
1. Request a GD subscription by calling:
NetEspSubscriberOptions.set_gd_session(string session_name
and creating a NetEspSubscriber object.
2. Create and connect a NetEspPublisher object.
3. Retrieve the checkpoint sequence number for the last checkpointed data by calling:
NetEspSubscriberEvent.get_checkpoint_sequence_number(NetEspError^ error).
4. Tell the server that the subscriber has committed messages up to a given sequence number and no longer needs them by calling:
NetEspPublisher.commit_gd(String^ session_name, array<int32_t>^ stream_ids, array<int64_t>^ seq_nos, NetEspError^ error).
5. Cancel the GD session to free up resources once it is no longer needed by closing the subscriber or by calling:
NetEspProject.cancel_gd_subscriber_session(String^ gd_session, NetEspError^ error).
Take a look at the SDK Guide for a more complete example of using guaranteed delivery with SDK subscribers.
For more information on working with Guaranteed Delivery and Consistent Recovery:
- Zero Data Loss
- Using Bindings to Connect CCL Projects
- Data Retention and Recovery with Stores
- See the Adapters guide for using adapters with Guaranteed Delivery.
- SAP Managed Tags:
