Exactly Once in SAP Hana Cloud Integration
Written by: Martin Jaspers
In one of our first proof of concepts with SAP Hana Cloud Integration (HCI) we noticed that, when a message failed in HCI, you couldn’t restart the message in HCI and it had to be restarted/resent at the source system.
When we went one step further and added a second receiver to the scenario with the multicast (in parallel mode) operation, we found out that, when the message to the first receiver went successful and the message to the second receiver failed, the message could not be restarted in HCI or the source system. It was not possible to guarantee the delivery to the second receiver.
We have discussed this with SAP and they confirmed that HCI (version 2.14.6) has no Quality of Service (QoS) Exactly Once (EO) out of the box, but it will be part of the HCI roadmap.
Together with SAP we developed an EO pattern in HCI with the help of the HCI Data Store. The iFlow consists of four parts:
- The Main Integration Process for the normal processing of the incoming message.
- A local Integration Process to send the message to the receiver, including error handling in case of an error.
- A second Integration Process to retrieve the message from the data store after the occurrence of an error.
- Second local Integration Progress to send the message to the receiver and delete the message on success from the Date Store
Part 1: The Main Integration Process
The main integration process has the normal processing operations, like Content Modifiers and Mappings. The last Content Modifier operation will set the dynamic file name (header variable “CamelFileName”) and will save the payload of the message in a property. The last operation is a Process Call to a local integration process. In this case it is a single receiver; in case of multiple receivers you will need a Multicast operation (in serial mode) with each a call to separate local integration process.
Part 2: The first local Integration Process
The first local Integration Process contains a send operation (“Service Call” operation in “Send” mode) and an Exception SubProcess. The Exception Subprocess will catch a potential error during the send step. The Content Modifier will make sure the message that has failed, is put back in the body of the message using the property set by the last Content Modifier operation in the main process. The Write operation (Data Store Operations in “Write” mode) will persist the body of the message to the HCI Data Store and the file name (as it is dynamic we don’t want to lose it) as Entry ID.
Part 3: The Retry Sending failed message
The third part is the “Retry sending failed messages” Integration Process within the same iFlow. This is a timed event that will trigger every 5 minutes a Select operation (Data Store Operations in “Select” mode).
This operation is configured to retrieve 100 message at the time and the checkbox “Delete After Fetch” is unchecked. The Select Operation will add a small envelop to the message:
The next step, a Router operation, will check if there are any messages retrieved from the data store with in our case the following Xpath expression:
count(/messages/message/OutputXMLReplicationRequest/DocumentType) > 0
If you don’t do this, you can process an empty message.
As we can retrieve a potential 100 messages at the same time from the data store, we need a Splitter operation to split up the messages and process them one by one.
The Content Modifier sets header variable “CamelFileName” for the dynamic file name. Since we used the file name as the Entry ID in the Data Store we can retrieve it with the following Xpath expression:
The last step before sending the message to the receiver via the local Integration Process is a Filter operation to remove the envelop from the message with in our case the following Xpath expression:
Part 4: The second local Integration Process
The messagewill be resend from a second local Integration Process. If the message has been sent successfully, the message will be deleted from the data store by the Delete operation (Data Store Operations in “Delete” mode). If the message fails again, The Delete operation will not be executed, the message will stay in the Data Store and will be processed again in the next occurrence of the Retry Integration Process.
We hope this Exactly Once Pattern will help you to build more robust interfaces in SAP Hana Cloud integration.
Frank & Martin.
Thanks for this blog. I had for long been wondering about the same Exactly Once topic and had come to the conclusion that this is not supported after multiple permutations and combinations of trying various adapters. Your blog confirms the same and I am glad this is on HCI's Roadmap. Am hoping this comes out ASAP! 🙂
This is a excellent blog and good thought process means to achieve the ability to re-send a message from HCI in case of communication errors. Excellent!
If I may suggest, it would also be great if you can include the details of each step type, especially the content modifier steps where you set the payloads, headers etc and also show a execution of this flow. This can be a separate blog but will definitely help the community!
Nice blog Frank ...hope more coming from you in future 🙂
Thank you for sharing the solution 🙂
In case we don't develop the solution, if the message fails in HCI it will wait for the next message processed from the transactional system, right?
So in case of failure the message is lost in HCI, not being reprocessed?
Excellent blog and thanks for sharing Frank and Martin.
It seems like Part 3 is no longer working. I have tried this and I couldn’t deploy the iFlow because of the following error:
“Direct call process should have a Start Event”
Can anyone confirm?
Thanks This blog is very useful.
Yes, thanks !!!