Skip to Content
Technical Articles
Author's profile photo Arundathi Sarala

Message retry using Process Direct and Data Store in SAP CPI

Introduction:

Currently CPI version 6.20.20 doesn’t have inbuilt functionality to reprocess failed asynchronous messages. For the continuous business support in maintenance projects, it’s important to handle the failures (connection failures, message failures). There are many blogs available on retry using JMS queues, but with the limitation on the JMS queues for each tenant its more challenging to implement the solution in real time. I would like to share an approach to implement the message retry using process direct adapter and data store.

Scenario:

Store the failed message details with payload, message status (flag which indicates the stage where message failed while processing), process direct endpoint of actual business logic iflow in data store. Schedule an iflow which picks the message from the data store and process it again. So, in further Iflows we shall see how to handle both connection errors and mapping errors.

To achieve this, I have divided the solution in 3 parts:

  1. Passthrough iflow
  2. Actual business logic iflow
  3. Retry iflow

Integration Artifact Details: iFlow 1: Passthrough iflow

This iflow will receive the message from source system. Here you can perform some basic conversions (json to xml), log source payload if required and add custom message search which will be helpful to monitor the message flow. For example, if you are receiving an IDOC, you can add the IDOC number in custom message search. You can either use a groovy script or use SAP_ApplicationID in content modifier- message header.

Process Direct Configuration:

Integration Artifact Details: iFlow 2: Actual business logic iflow

In this iflow implement the actual business logic. I will not explain more on business logic used in above flow, as our focus in on the parameters required for message retry.
Step 1: Process Direct
Reuse the process direct path used in passthrough iflow

Step 2: Custom Msg Search
In this script save the input payload, set the initial message status to “MapFail” as this step is performed before mapping and add custom header.

import com.sap.gateway.ip.core.customdev.util.Message;

def Message processData(Message message) {
    def body = message.getBody(java.lang.String) as String;
    def parse = new XmlParser().parseText(body);
	def messageLog = messageLogFactory.getMessageLog(message);
	    //Read initial property values
       map = message.getHeaders();
         def Parm1 = map.get("Parm1");
         def filename = map.get("Parm2");

	if(messageLog != null){
			message.setProperty("InputPayload", body);
			message.setProperty("MsgStatus", 'MapFail');
			message.setHeader("SAP_ApplicationID", Parm1);

	}
       
	return message;
}​

Step 3:
Perform the actual business logic which might include mappings, RFC calls, data conversions.

Step 4: Log MsgStatus

If the message fails due to mapping or connection issue, the exception sub-process will be invoked and below steps will be performed.

Step 5: Retrieve incoming message into body


Step 6: Log Msg Headers Payload
XSLT code to log msg headers and the payload. Extra parameters are defined if in case you need to pass more parameters.

<?xml version="1.0" encoding="UTF-8"?>
<xsl:stylesheet version="3.0" xmlns:xsl="http://www.w3.org/1999/XSL/Transform">
  <!--  <xsl:output method="xml" omit-xml-declaration="yes"/> -->
    <xsl:param name= "Endpoint"/>
    <xsl:param name= "EntryID"/>
    <xsl:param name= "MsgStatus"/>
    <xsl:param name= "Parm1"/>
    <xsl:param name= "Parm2"/>
    <xsl:param name= "Parm3"/>
    <xsl:param name= "Parm4"/>
    <xsl:param name= "Parm5"/>

    <xsl:template match="node()|@*">
        <xsl:copy>
            <xsl:apply-templates select="node()|@*"/>
        </xsl:copy>
    </xsl:template>
    
        <xsl:template match="/">
            <RootNode>
                <Payload>
                   <xsl:apply-templates select="node()|@*"/>                 
                </Payload>
                <MsgHeader>
                    <Endpoint><xsl:value-of select="$Endpoint"/></Endpoint> 
		            <EntryID><xsl:value-of select="$EntryID"/></EntryID>
	        	    <MsgStatus><xsl:value-of select="$MsgStatus"/></MsgStatus>
	        	    <Parm1><xsl:value-of select="$Parm1"/></Parm1>
	        	    <Parm2><xsl:value-of select="$Parm2"/></Parm2>
	        	    <Parm3><xsl:value-of select="$Parm3"/></Parm3>
	        	    <Parm4><xsl:value-of select="$Parm4"/></Parm4>
	        	    <Parm5><xsl:value-of select="$Parm5"/></Parm5>
		        </MsgHeader>
              </RootNode>
        </xsl:template> 
</xsl:stylesheet>

Step 7: Backup
Save the information generated in step 6 in data store “DS_RetryAllError”. EntryID can be of your choice, or you can leave it blank. Here I have used message ID. Note here all the connection and message error details are saved in common “DS_RetryAllError”.

Integration Artifact Details: iFlow 3: Retry iflow – For all errors

Picks all the messages from data store “DS_RetryAllError” if the message had failed earlier due to connection issue pushes file to corresponding iflow. If mapping issue it is sent to data store “DS_RetryMapError”.

       Step 1:Start Timer: Schedule the interface to recur daily with polling interval every           minute. 

       Step 2:Select Message:  Pick a file from data store “DS_RetryAllError”

      Step 3:Router: Processes ahead if any file present in datastore or else ends the process           execution.

        Step 4:Get Header Parameters: Retrieve the process direct path and reason for                         message failure (MsgStatus)

         Step 5:Router: If the message failed due to mapping error move the data to new data       store = “DS_RetryMapError” if not retrieve only the payload and pass it to actual business iflow.

    Step 6:Filter Payload: Retrieve the payload

       Step 7:Process Direct:

        Step 8:Write Map Error DS: Store the message errored due to mapping issue to data store =“DS_RetryMapError”.

Integration Artifact Details: iFlow 4: Retry iflow – For all mapping errors

Mapping errors can occur due to some data issues or mapping logics implemented due to business requirement. Logical errors can be fixed in some scenarios and in such case, we can retry the payload stored in “DS_RetryMapError” using below iflow after the mapping issue is fixed in main iflow.

Picks the mentioned message in entryID from data store “DS_RetryMapError” and pushes file to corresponding iflow. This iflow runs on demand after the identified mapping issue has been fixed in actual iflow.

Step 1:Start Timer: Run Once

Step 2:Get Message:  Pick a file from data store “DS_RetryMapError” with specified entry ID

Step 3:Get Header Parameters: Retrieve the process direct path and other Parameters if required.

Step 4:Filter Payload: Retrieve the payload

Step 5:Process Direct:

Message failed due to data issue cannot be retried and need to be deleted from data store “DS_RetryMapError”. For this you can schedule an iflow which is explained in blog shared in reference section.

Conclusion:
We just saw how to configure the retry mechanism to handle both connection and mapping failures. The above retry mechanism can be reused across all the asynchronous interfaces built in a tenant.

References:

https://blogs.sap.com/2021/10/29/automatic-data-store-cleanup-using-sap-api/

Assigned Tags

      21 Comments
      You must be Logged on to comment or reply to a post.
      Author's profile photo Lokesh Tarey
      Lokesh Tarey

      Well explained. Very eloquent and descriptive.

      Author's profile photo Syambabu Allu
      Syambabu Allu

      Hi Arundathi,

      Thanks for sharing..it’s really helpful blog.

      Thank you,

      Syam

      Author's profile photo Arundathi Sarala
      Arundathi Sarala
      Blog Post Author

      Thank you Lokesh and Syam 😊

      Author's profile photo rajesh kumar
      rajesh kumar

      Very Insightful & well Explained !!!

      Author's profile photo Wilson Xavier
      Wilson Xavier

      Fantastic, thanks a lot!

      Author's profile photo Arundathi Sarala
      Arundathi Sarala
      Blog Post Author

      Thank you Rajesh and Wilson 🙂

      Author's profile photo Andrey Tkachuk
      Andrey Tkachuk

      Why create an instance of a class "messageLog" if its methods have never been used anywhere?

      def messageLog = messageLogFactory.getMessageLog(message);

       

      What guarantees does verification of its creation provide?

      if(messageLog != null){
      Author's profile photo Arundathi Sarala
      Arundathi Sarala
      Blog Post Author

      Hello Andrey,

      The MessageLogFactory class is used to create the messageLog object.
      The addCustomHeaderProperty function can then be used to add custom header fields.
      There was a business requirement to add one such header in actual interface. For simplicity I have removed all the business logics from the code.
      The attempt here is to set the MsgStatus along with custom message headers script. You can either set the message status in a script or just use a content modifier.

      Please refer the below link for more details on custom message header and messageLog.https://help.sap.com/viewer/368c481cd6954bdfa5d0435479fd4eaf/Cloud/en-US/d4b5839670ce4866a770f7cadac063db.html

       

      Author's profile photo Wilson Xavier
      Wilson Xavier

      Arundathi Sarala Could you share the sample for download?

      Author's profile photo Arundathi Sarala
      Arundathi Sarala
      Blog Post Author

      Hey Wilson,

      Sorry I had implemented this in one of the client system. So cannot share any downloads.

      I have explained each stage in blog clearly, please follow the same. If you are stuck at any stage or need any support, I would be very happy to help you.

      Thank you!!

       

      Author's profile photo Eniyan Thangaraja
      Eniyan Thangaraja

      Hi Arundathi,

       

      That's a nice blog! I have a question. When we have a timer in place for retries, we don't have any control on the number of retries. Is there a way where we can limit the number of retries?

       

      For Example, I don't want the retries to happen more than 3 times for the same message. Any insights here?

       

      Regards,

      Eniyan.

      Author's profile photo Philippe Addor
      Philippe Addor

      I would say so too. Otherwise the process gets blocked and subsequent messages are not handled. See also my reply further below.

      Author's profile photo Arundathi Sarala
      Arundathi Sarala
      Blog Post Author

      Hello Eniyan,

       

      Apologies for the late response.

      As Philippe suggested you can have a retry counter to limit the number of retries. In my case there wasn't any business requirement for retry limit so didn't implement it.

      Thank you!

      Author's profile photo David Da Silva
      David Da Silva

      Hi Arundathi Sarala,

      Thank you so much for writing this article! Very informative and it helped me with a challenge I was facing.

      Best regards,

      David

      Author's profile photo Philippe Addor
      Philippe Addor

      Clever approach! Thanks for sharing it.

      However, I think you need to add a retry counter to limit the number of retries in case of permanent connection errors. If I'm not mistaking, the following would happen in the described implementation: Although the retry Iflow deletes the entry every time a retry is done (due to the setting in the read step), in case of permanent errors in the business logic Iflow, that one would keep creating new entries in the AllErrors data store. During the next run of the retry Iflow it would again pick up the same message and the whole thing ends up in an endless loop. This means that subsequent error messages would not be handled. To overcome this, I suggest to add a counter value to the header structure that is persisted in the data store, and when reading it and the value is higher than e.g. 2, escalate the message instead of retrying it. 

      On a side note: please name your routes (of the Router steps) with meaningful names for sake of readability. And step boxes can be increased in size so that the text is fully readable. Also, Iflow steps are ideally placed with a grid-like alignment with straight lines and where possible with a left to right flow. See also in the Design Guidelines: https://help.sap.com/docs/CLOUD_INTEGRATION/368c481cd6954bdfa5d0435479fd4eaf/32e4f6e216cb4e11b1c151c5bf538224.html

      Author's profile photo Arundathi Sarala
      Arundathi Sarala
      Blog Post Author

      Hi Philippe,

      Thank you for the inputs on naming the route steps and the design guidelines!!

      The message retry was created for connection errors and mapping errors. Connection errors are not permanent and as per the business requirement we wanted to retain the messages in "DS_RetryAllError" after subsequent connection failures. As suggested by you above, you can have a retry counter to limit the number of retries for connection errors and move it to another data store. Once the connection is established you can process the messages from the data store.

      As per my understanding mapping errors might be either due to conversion logics( UDFs) or due to incorrect data from source. So such messages are moved to data store “DS_RetryMapError”. After fixing the mapping logic you can retry the data store entry using "iFlow 4: Retry iflow – For all mapping errors". Messages failed due to data issue which cannot be resent from CPI will remain in data store. This data store can be cleared using the approach shared in my other blog below.

      https://blogs.sap.com/2021/10/29/automatic-data-store-cleanup-using-sap-api/

      If you have faced any other permanent business errors which are not handled in above design kindly share them. It would be interesting to find solution for it as well. 🙂

      Thank you!

      Author's profile photo Yeswanth Posam
      Yeswanth Posam

      Hi Philippe,

      Unfortunately Datastore Select operation is not fetching headers included during Datastore Write operation. Please let me know if you found any alternate approach to this.

      Author's profile photo Philippe Addor
      Philippe Addor

      Hi  Yeswanth Posam,

      That is true, but I was not talking about the message header but about the header structure that Arundathi included in her message structure which holds the header and the payload of the original message. See step 6 above.

      I mean this node:

      <MsgHeader>

      Best regards,

      Philippe

      Author's profile photo Yeswanth Posam
      Yeswanth Posam

      Thankyou Philippe Addor for the quick response.

      We are using Datastore sender adapter for Retry mechanism which is having SAP_DataStoreRetries header for the retry count. It is also fetching other Iflow specific headers which are included as part of Datastore Write operation.

      Author's profile photo Philippe Addor
      Philippe Addor

      You're right, the DataStore Sender Adapter makes it possible to retrieve the headers if "Allowed Headers" is configured in the Iflow Runtime Configuration! I wasn't aware of that neither. This is great to know.

      Thanks for sharing your approach.

      Author's profile photo Sankara Bhatta
      Sankara Bhatta

      HI,

      Very useful information. Do you know if SAP is planning to have this retry feature as part of standard platform in future. it is really funny that a middleware does not have this capability. SAP solutions like SAP PI/PO have this functionality