Managing time intensive Integration Flows using SAP Cloud Platform Integration Suite

Problem Statement:

Integration Flows(iFlow) can be used to solve problems which are time intensive(the job takes ‘long’ time to end).Though there are necessary measures in place within Cloud Platform Integration Suite to handle such time intensive processes, if the endpoint of the iFlow is managed by an API Management layer then the long lived connection is a challenge to manage. This is because REST/OData architecture preaches short lived request-responses.iFlows in this case is long-lived. Therefore any request coming from the proxy endpoint can result in timeouts.

In this blog I am going to talk about two architecture patterns which can be used to solve this problem

Webhook Pattern:

The Flow:

When deployed the iFlow subscribes to a Queue within SAP Enterprise Messaging. An API is created in APIM system which is configured to publish to the Queue which the iFlow has subscribed.

1. The client requests the API endpoint exposed with the payload the webhook URL.
2. On receive of the request, APIM executes the policies provided and creates a job-publishes the payload received from the client in the Queue.
3. It returns a response to the client as – a job has been created. Once the job is finished the webhook would be called.
4. The iFlow is subscribed for jobs found in the queue
5. iFlow starts processing the time intensive job.
6. iFlow requests to several different systems to finish the job
7. iFlow collects all the responses
8. Once the job ends it extracts the webhook URL provided by the client and posts the response to the webhook URL.

Key points to consider:

1. The client has to own the response payload . Any downtime at the client’s end can cause unavailability for the webhook URL . In such a case the job response payload might get lost.
2. The request goes through the API managed gate, the response doesn’t go through any managed gate. This could lead to security issues for some organisation. How trusted is the webhook ?

LookUp Pattern:

The Flow:

When deployed the iFlow subscribes to a Queue within SAP Enterprise Messaging.There are two queues Request&Response. An API is created in APIM system which is configured to publish to the Queue which the iFlow has subscribed.

1. The client requests the API endpoint .
2. On receive of the request, APIM executes the policies provided and creates a job-publishes the payload received from the client in the Queue.
3. It returns a response to the client as – a job has been created.
4. The iFlow is subscribed for jobs found in the queue
5. iFlow starts processing the time intensive job.
6. iFlow requests to several different systems to finish the job.
7. iFlow collects all the responses
8. iFlow queues the job-completion status within the Response Queue.
   1. New job-status is found in the Queue
   2. An iFlow processes the completion status and stores the response within a Data-Store. Note- From Step-7 there could be direct insertion into the data-store as well. The response Queue might not be required at all, however in some cases its best practise to use a queue for ‘job’ handling.
9. The client looks-up for the response . The client could poll in this case.
10. Job completion Status is fetched from the Store.
11. The  completion response is returned to the client.

Key points to consider:

1. The provider is the owner for the storage of the response.How long will the response be stored? When and How will the response be deleted ? depends on the requirement and the landscape
2. The client can poll to know the status of the job. This could impact the server as needless traffic be consumed due to polling.

Both patterns

• ensure that there is a guarantee of response to the client – inbound request- response.
• are asynchronous with respect to the client.
• are proven to scale incase the job completion time increases or decreases

Edit: I am getting questions when to use this pattern :

Use this pattern when APIM is used with CPI with the following OR conditions

• Completion of iFlows is going to take a lot of time. This is already known in advance.
• Completion time of iFlows is unknown .Time taken by the iFlow may be long or short depending on the scenario.
• Want a generic architecture which should be able to scale handling of both short(immediate) and long running flows in future

Please note for already known short running iFlows this architecture could be an overkill.