Simply implement retry mechanism in SAP CPI without JMS

[stephen_xue]

Introduction

One of the pain points of using SAP CPI is that it has not out of box function of retry mechanism as that in SAP PI/PO for asynchronous communication. I found there are quite a few blogs introducing how to fultill retry by using JMS queue in SAP CPI. Since we know that there is a number limitation of JMS queues provided for each tenant, I'd like to provide a way implementing retry without JMS.

The whole solution consists of two main parts:

1. Payload Backup from Exception;


2. Retrieve Payload for retry;

In case you are not interested in the technical details of the contens, just download the iflows from here. And import it to your package.

Implementation Steps

Firstly, Payload Backup from Exception

The overall iFlow is as below. To make the logic easy to understand, all of the redundent steps have been removed from the flow.

For step 1 and 4. Please ignore the sender/receiver adapters used in the iflow. They are here for elaborating only. They can be replaced by any adapters in the asynchronous process, like https, XI, JMS, JDBC etc.

For step 2. Please add your own mapping step or content modified step or event a sub process before the payload has been backed up.

For step 3. Backup the payload to message property so that once the exception occurred, the payload can be get back. This is how the payload has been backeup for the convenience of new developers.

For step 5. Once exception sub process has been triggerred, the original payload will be refreshed. This step is to get the original payload back.

This is how the step has been configured

meanwhile, in order to let the framework generate a random ID for the data store entry id, this step is needed as well in the step deleting the http header property: SapDataStoreId

This is the reason stading by help.sap.com:

https://help.sap.com/viewer/368c481cd6954bdfa5d0435479fd4eaf/Cloud/en-US/46260ee193ed48f48a9bd5b8558...

For step 6. Backup the message body into a global data store. This is the configuration.

As explained in the previous step, the field 'Entry ID' is blank.

Please provide a name for the data store making sure the name is process specific. For example

• GL_DS_SALES_ORDER_CREATE


• GL_DS_MATERIAL_MASTER_UPDATE

and leave the visibility as 'Global'.

 

Secondly, Retrieve stored payload for retry.

This is this the overview of the process

For step 1. It is a timer event. configure the start time, frequency to retrieve the failed message from global data store.This is how it has been configured in my example

For step 2. In the step, the backup message has been retrieved in random order from the global data store. Please be aware that the select operation of data store cannot guarentee the sequence. This mechanism has been addressed in help.sap.com:

https://help.sap.com/viewer/368c481cd6954bdfa5d0435479fd4eaf/Cloud/en-US/8cfe004231bd45d9bdf497eced8...

There is no in-order processing of data store entries. In other words, the sequence of data store entries selected with a Select operation is random and non-deterministic. If you have restricted the number of entries to be fetched (with the Number of Polled Messages attribute), the selection of retrieved entries is also random.

specify the data source name which is exactly the same as step 6 of First part.

The yellow field provides an option to read the data source in parellel.

Check the flag 'Delete On Completion' so that the message can be removed once it has been pushed to the subsequential system to avoid duplication.

For step 3. Check whether message retrieved from the data source is empty or not. The select operation will keep retrieve until the timer event finished. If all of the messages inside the data source have been retrieved, the select operation will generate an empty message as below

<?xml version="1.0" encoding="UTF-8"?>

<messages/>

To avoid redundent call made to down stream system, a router with bellowing xpath condition can be added making the process more elegant.

/messages[not(message)]

For step 4. Since the select operation will wrap up the original message with an outer element automatically, to recover the message, an XSLT mapping has been used to strip it off. This is the source code.

<xsl:stylesheet version="3.0"   xmlns:xsl  ="http://www.w3.org/1999/XSL/Transform">

    <xsl:output method="xml" version="1.0" encoding="UTF-8" indent="yes"/>

    <!--********************************************************************************************** -->

    <!--Program: CPI_GetPayloadFromDataStore.xsl                                                       -->

    <!--Create Date: 26-Nov-2020                                                                       -->

    <!--Author: Stephen Xue                                                                            -->

    <!--Description:                                                                                   -->

    <!--Remove wrap up element from data store                                                         -->

    <!--***********************************************************************************************-->

    <xsl:template match="/">

        <xsl:copy-of select="messages/message/*"/>

    </xsl:template>

</xsl:stylesheet>

 

For Step 5. Push the message to downstream system.

The configuration should be the same as step 4 of First part.

For step 6. To ensure the message be consistent, if the data pushing process to the downstream system failed again, which is step 5 here, the data retrieving process from the data store, which is step 2 here,  should be rollback in the same transaction. Therefore, make sure the Transaction handling is 'Required for JDBC' for this whole iflow process. Actually, by default, this has been configured.

Conclusion

For each tenant, the data store has maximum 32G capacity and if you are lack of JMS resource, please feel free to download it here and take it as an reference.