SAP Integration Suite: Resilient APIs using API Management, Event Mesh, and Cloud Integration
This blog article outlines an idea how to design APIs and interfaces which are more robust towards unavailable services, interface errors, and data loss. The following services of SAP Integration Suite are used to achieve this goal: SAP API Management, SAP Event Mesh, and SAP Cloud Integration. Similar ideas for applications are described in the discovery center mission “Enhance core ERP business processes with resilient applications on SAP BTP“: https://discovery-center.cloud.sap/missiondetail/3501/3542/
In this scenario, sales orders are received via a public and asynchronous API, delivered to the event mesh, and transmitted via cloud integration to SAP S/4. The problem addressed here is how to reduce data loss when running into situations like (1) errors in the request payload leading to mapping errors or errors in SAP S/4 processing, (2) unavailability of the iFlow for example when redeploying it, or (3) unavailability of the receiver system SAP S/4.
Of course, there is more than one solution to this, and I am sure I haven’t seen them all. Apart from the solution presented here you could for example:
- Archive events in the data store of the cloud integration or in a cloud storage outside of the Integration Suite. This only works if the IFlow is always operational.
- Use the retry functionality of the XI Adapter. All events triggering an error before reaching the XI adapter still cannot be reprocessed and are lost.
- Use a CAP service within a data integration hub. You get persistence of data for free and comfortable validation, plus, you are well prepared for clean core and side-by-side architectures which will be relevant in the future as described here by Thorsten. But you need the skill to implement and maintain CAP services.
Do you have any other ideas? Please let me know in the comments.
The following aspects of this scenario are outlined in six steps:
- Configuration of the event mesh instance.
- Configuration of the iFlow.
- Some validation capabilities of the API.
- Reprocessing events in case of errors.
Step 1: Create an event mesh instance
Set up an event mesh instance in your integration subaccount and create two keys – one for the API and one for the Cloud Integration access, for example:
Step 2: Create the queues
Once the event mesh instance is up and running, create queues. One for processing events and one dead message queue:
Configure the main queue for processing events with a dead message option:
Step 3: Create the IFlow
This iFlow pulls events from the event mesh queue via AMQP adapter, performs a json2xml conversion, a message mapping using groovy and delivers the request via XI adapter to SAP S/4. Alerts are sent via exception process to SAP AIF for better monitoring.
Here is how you would configure the AMQP adapter:
Details of how to utilize the AMQP adapter in cloud integration are described in this excellent blog:
Cloud Integration – Connecting to Messaging Systems using the AMQP Adapter by Mandy Krimmel
Step 4: Expose the API
Create, configure, and publish your API in the API Hub Portal.
Call the event mesh instance directly by entering the URL as target endpoint.
Note: In case the API provides more than one REST resource, the resource name will be added to the URL by default when calling the target endpoint. We chose to rewrite the target URL in the API policy:
Here are references for this step:
Step 5: Test – sending sample requests
Below you see the response after we have sent an order request to the API. Since the API is asynchronous and moves the events into a queue of the event mesh, the response only states that data has been received and will be processed. At this point, we can not respond with a proper information about the real status of the processing of the order.
- Success – custom response and return code 201
- Failure – responding with validation error 422
To reduce errors in the IFlow and the backend system we have set up some validation for incoming requests – here is an example with a wrong date format and a missing mandatory field. This request is blocked right away and the sender receives a verbose response:
Step 6: Reprocessing orders
Once the events have passed the API validation and there is an error in the cloud integration, events are moved to the dead message queue. There are different ways how to handle this, we decided to simply use an additional small IFlow which moves all events from the dead message queue back to the orders queue. As soon as the error in the cloud integration is fixed, the IFlow will deliver the events to SAP S/4 again. In case the IFlow is undeployed or not operational, events remain in the orders queue and are automatically processed once the IFlow is deployed. If the receiver SAP S/4 is not available, requests can be stored in the JMS queue of the XI adapter for reprocessing.
This blog post provides some ideas how to create more resilient APIs. An use case is presented as an implementation utilizing a combination of SAP API Management, SAP Event Mesh and SAP Cloud Integration.
Please feel free to post a query in the comments section if you have any inquiries.