Technical Articles
Message retry using Process Direct and Data Store in SAP CPI
Introduction:
Currently CPI version 6.20.20 doesn’t have inbuilt functionality to reprocess failed asynchronous messages. For the continuous business support in maintenance projects, it’s important to handle the failures (connection failures, message failures). There are many blogs available on retry using JMS queues, but with the limitation on the JMS queues for each tenant its more challenging to implement the solution in real time. I would like to share an approach to implement the message retry using process direct adapter and data store.
Scenario:
Store the failed message details with payload, message status (flag which indicates the stage where message failed while processing), process direct endpoint of actual business logic iflow in data store. Schedule an iflow which picks the message from the data store and process it again. So, in further Iflows we shall see how to handle both connection errors and mapping errors.
To achieve this, I have divided the solution in 3 parts:
- Passthrough iflow
- Actual business logic iflow
- Retry iflow
Integration Artifact Details: iFlow 1: Passthrough iflow
This iflow will receive the message from source system. Here you can perform some basic conversions (json to xml), log source payload if required and add custom message search which will be helpful to monitor the message flow. For example, if you are receiving an IDOC, you can add the IDOC number in custom message search. You can either use a groovy script or use SAP_ApplicationID in content modifier- message header.
Process Direct Configuration:
Step 2: Custom Msg Search
In this script save the input payload, set the initial message status to “MapFail” as this step is performed before mapping and add custom header.
import com.sap.gateway.ip.core.customdev.util.Message;
def Message processData(Message message) {
def body = message.getBody(java.lang.String) as String;
def parse = new XmlParser().parseText(body);
def messageLog = messageLogFactory.getMessageLog(message);
//Read initial property values
map = message.getHeaders();
def Parm1 = map.get("Parm1");
def filename = map.get("Parm2");
if(messageLog != null){
message.setProperty("InputPayload", body);
message.setProperty("MsgStatus", 'MapFail');
message.setHeader("SAP_ApplicationID", Parm1);
}
return message;
}
Step 4: Log MsgStatus
If the message fails due to mapping or connection issue, the exception sub-process will be invoked and below steps will be performed.
<?xml version="1.0" encoding="UTF-8"?>
<xsl:stylesheet version="3.0" xmlns:xsl="http://www.w3.org/1999/XSL/Transform">
<!-- <xsl:output method="xml" omit-xml-declaration="yes"/> -->
<xsl:param name= "Endpoint"/>
<xsl:param name= "EntryID"/>
<xsl:param name= "MsgStatus"/>
<xsl:param name= "Parm1"/>
<xsl:param name= "Parm2"/>
<xsl:param name= "Parm3"/>
<xsl:param name= "Parm4"/>
<xsl:param name= "Parm5"/>
<xsl:template match="node()|@*">
<xsl:copy>
<xsl:apply-templates select="node()|@*"/>
</xsl:copy>
</xsl:template>
<xsl:template match="/">
<RootNode>
<Payload>
<xsl:apply-templates select="node()|@*"/>
</Payload>
<MsgHeader>
<Endpoint><xsl:value-of select="$Endpoint"/></Endpoint>
<EntryID><xsl:value-of select="$EntryID"/></EntryID>
<MsgStatus><xsl:value-of select="$MsgStatus"/></MsgStatus>
<Parm1><xsl:value-of select="$Parm1"/></Parm1>
<Parm2><xsl:value-of select="$Parm2"/></Parm2>
<Parm3><xsl:value-of select="$Parm3"/></Parm3>
<Parm4><xsl:value-of select="$Parm4"/></Parm4>
<Parm5><xsl:value-of select="$Parm5"/></Parm5>
</MsgHeader>
</RootNode>
</xsl:template>
</xsl:stylesheet>
Integration Artifact Details: iFlow 3: Retry iflow – For all errors
Picks all the messages from data store “DS_RetryAllError” if the message had failed earlier due to connection issue pushes file to corresponding iflow. If mapping issue it is sent to data store “DS_RetryMapError”.
Step 1:Start Timer: Schedule the interface to recur daily with polling interval every minute.
Step 2:Select Message: Pick a file from data store “DS_RetryAllError”
Step 3:Router: Processes ahead if any file present in datastore or else ends the process execution.
Step 4:Get Header Parameters: Retrieve the process direct path and reason for message failure (MsgStatus)
Step 5:Router: If the message failed due to mapping error move the data to new data store = “DS_RetryMapError” if not retrieve only the payload and pass it to actual business iflow.
Step 6:Filter Payload: Retrieve the payload
Step 7:Process Direct:
Step 8:Write Map Error DS: Store the message errored due to mapping issue to data store =“DS_RetryMapError”.
Mapping errors can occur due to some data issues or mapping logics implemented due to business requirement. Logical errors can be fixed in some scenarios and in such case, we can retry the payload stored in “DS_RetryMapError” using below iflow after the mapping issue is fixed in main iflow.
Picks the mentioned message in entryID from data store “DS_RetryMapError” and pushes file to corresponding iflow. This iflow runs on demand after the identified mapping issue has been fixed in actual iflow.
Step 1:Start Timer: Run Once
Step 2:Get Message: Pick a file from data store “DS_RetryMapError” with specified entry ID
Step 3:Get Header Parameters: Retrieve the process direct path and other Parameters if required.
Step 4:Filter Payload: Retrieve the payload
Step 5:Process Direct:
Message failed due to data issue cannot be retried and need to be deleted from data store “DS_RetryMapError”. For this you can schedule an iflow which is explained in blog shared in reference section.
Conclusion:
We just saw how to configure the retry mechanism to handle both connection and mapping failures. The above retry mechanism can be reused across all the asynchronous interfaces built in a tenant.
References:
https://blogs.sap.com/2021/10/29/automatic-data-store-cleanup-using-sap-api/
Well explained. Very eloquent and descriptive.
Hi Arundathi,
Thanks for sharing..it’s really helpful blog.
Thank you,
Syam
Thank you Lokesh and Syam 😊
Very Insightful & well Explained !!!
Fantastic, thanks a lot!
Thank you Rajesh and Wilson 🙂
Why create an instance of a class "messageLog" if its methods have never been used anywhere?
What guarantees does verification of its creation provide?
Hello Andrey,
The MessageLogFactory class is used to create the messageLog object.
The addCustomHeaderProperty function can then be used to add custom header fields.
There was a business requirement to add one such header in actual interface. For simplicity I have removed all the business logics from the code.
The attempt here is to set the MsgStatus along with custom message headers script. You can either set the message status in a script or just use a content modifier.
Please refer the below link for more details on custom message header and messageLog.https://help.sap.com/viewer/368c481cd6954bdfa5d0435479fd4eaf/Cloud/en-US/d4b5839670ce4866a770f7cadac063db.html
Arundathi Sarala Could you share the sample for download?
Hey Wilson,
Sorry I had implemented this in one of the client system. So cannot share any downloads.
I have explained each stage in blog clearly, please follow the same. If you are stuck at any stage or need any support, I would be very happy to help you.
Thank you!!
Hi Arundathi,
That's a nice blog! I have a question. When we have a timer in place for retries, we don't have any control on the number of retries. Is there a way where we can limit the number of retries?
For Example, I don't want the retries to happen more than 3 times for the same message. Any insights here?
Regards,
Eniyan.
I would say so too. Otherwise the process gets blocked and subsequent messages are not handled. See also my reply further below.
Hello Eniyan,
Apologies for the late response.
As Philippe suggested you can have a retry counter to limit the number of retries. In my case there wasn't any business requirement for retry limit so didn't implement it.
Thank you!
Hi Arundathi Sarala,
Thank you so much for writing this article! Very informative and it helped me with a challenge I was facing.
Best regards,
David
Clever approach! Thanks for sharing it.
However, I think you need to add a retry counter to limit the number of retries in case of permanent connection errors. If I'm not mistaking, the following would happen in the described implementation: Although the retry Iflow deletes the entry every time a retry is done (due to the setting in the read step), in case of permanent errors in the business logic Iflow, that one would keep creating new entries in the AllErrors data store. During the next run of the retry Iflow it would again pick up the same message and the whole thing ends up in an endless loop. This means that subsequent error messages would not be handled. To overcome this, I suggest to add a counter value to the header structure that is persisted in the data store, and when reading it and the value is higher than e.g. 2, escalate the message instead of retrying it.
On a side note: please name your routes (of the Router steps) with meaningful names for sake of readability. And step boxes can be increased in size so that the text is fully readable. Also, Iflow steps are ideally placed with a grid-like alignment with straight lines and where possible with a left to right flow. See also in the Design Guidelines: https://help.sap.com/docs/CLOUD_INTEGRATION/368c481cd6954bdfa5d0435479fd4eaf/32e4f6e216cb4e11b1c151c5bf538224.html
Hi Philippe,
Thank you for the inputs on naming the route steps and the design guidelines!!
The message retry was created for connection errors and mapping errors. Connection errors are not permanent and as per the business requirement we wanted to retain the messages in "DS_RetryAllError" after subsequent connection failures. As suggested by you above, you can have a retry counter to limit the number of retries for connection errors and move it to another data store. Once the connection is established you can process the messages from the data store.
As per my understanding mapping errors might be either due to conversion logics( UDFs) or due to incorrect data from source. So such messages are moved to data store “DS_RetryMapError”. After fixing the mapping logic you can retry the data store entry using "iFlow 4: Retry iflow – For all mapping errors". Messages failed due to data issue which cannot be resent from CPI will remain in data store. This data store can be cleared using the approach shared in my other blog below.
https://blogs.sap.com/2021/10/29/automatic-data-store-cleanup-using-sap-api/
If you have faced any other permanent business errors which are not handled in above design kindly share them. It would be interesting to find solution for it as well. 🙂
Thank you!
Hi Philippe,
Unfortunately Datastore Select operation is not fetching headers included during Datastore Write operation. Please let me know if you found any alternate approach to this.
Hi Yeswanth Posam,
That is true, but I was not talking about the message header but about the header structure that Arundathi included in her message structure which holds the header and the payload of the original message. See step 6 above.
I mean this node:
Best regards,
Philippe
Thankyou Philippe Addor for the quick response.
We are using Datastore sender adapter for Retry mechanism which is having SAP_DataStoreRetries header for the retry count. It is also fetching other Iflow specific headers which are included as part of Datastore Write operation.
You're right, the DataStore Sender Adapter makes it possible to retrieve the headers if "Allowed Headers" is configured in the Iflow Runtime Configuration! I wasn't aware of that neither. This is great to know.
Thanks for sharing your approach.
HI,
Very useful information. Do you know if SAP is planning to have this retry feature as part of standard platform in future. it is really funny that a middleware does not have this capability. SAP solutions like SAP PI/PO have this functionality
Thanks for your Blog. It is very useful for support projects.