SAP Enterprise Messaging and SAP Cloud Platform Integration (Pub/Sub Pattern)

[Jon1]

Background:

This blog post attempts to take a common scenario I’ve seen over several SAP Cloud Platform Integration projects which I hope others will find helpful. It was something that couldn't be achieved in the SAP PO days (easily) and I for one am happy that this is available in SAP Cloud Platform Integration:

• Multiple receivers of the one piece of data and/or Pub Sub

Whilst there is nothing stopping you from having 3 receivers of a single message in one iflow, you will run into issues if the first receiver is successful and subsequent ones fail. Eg how to restart this, how to prevent duplicates etc. You can somewhat mitigate this through data stores, queues within SAP Cloud Platform Integration, but would require bespoke development each time. Creating an iflow for each receiver is also cumbersome from a maintenance and creates unnecessary load. (eg each iflow calls SuccessFactors to get changes)

This blog aims to reduce development and utilize SAP Enterprise Messaging topics and queues to increase re-usability to include new consumers. I've used SuccessFactors as an example, but there are now S/4HANA events that will use this type of functionality, while any system that may wish to publish an event or fit in a pub/sub integration pattern may be applicable here.

Things to note:

• This functionality is separate to the inbuilt queues in the SAP Cloud Platform Integration Enterprise Edition


• This FAQ is a handy read (re: Licencing etc). Or of course contact your friendly Account folks

Pre-requisite for this blog:

• Knowledge of developing in SAP Cloud Platform Integration


• Getting Started with SAP Enterprise Messaging
  
  
  
  • (Activate the service in the SAP Cloud Platform trial account)
  
  
  • When specifying the parameters (json file) you can use the below. Details of the syntax are covered here. This will ensure Messaging and REST are active
  
  
  
  

Other useful reads:

• SAP EM - Message Broker for Events S4


• SAP EM and CPI for asynchronous patterns


• SAP Cloud Platform Enterprise Messaging – Making S/4HANA Event Notification Easy


• SAP Cloud Platform Enterprise Messaging as an eventbus for SuccessFactors
  
  
  
  • This could be an alternative to 'calling' SuccessFactors for changes
  
  
  
  

Scenario:

SuccessFactors employee changes are distributed to multiple systems.



High Level Steps:

• Setup SAP Cloud Platform – Enterprise Messaging (See pre-requisite blog)


• Create Topics and Queues


• Create SAP Cloud Platform Integration flow to write to the Topic


• Create SAP Cloud Platform Integration flow to read off the Queue and acknowledge the message

Please ensure you have activated the SAP Cloud Platform - Enterprise Messaging service beforehand (see pre-requisites)

Step 1: 

Log into SAP Foundry Trial and go to your Sub Account and then your Space



Step 2:

Within your space, click on your Enterprise Messaging instance



Step 3:

Navigate to Service Keys and keep this information for later in notepad (It will be needed for authentication). If one doesn’t exist, create one. This will show all the goodies to connect to the SAP Enterprise Messaging service through a variety of protocols



Step 4:

Click on Open Dashboard to take a peek at what's happening with the service



Step 5:

Now this is where we'll create our queues. It’s a bit counter intuitive for our scenario, but create Queues first. In this case, create: EmpReceiver1 and EmpReceiver2.



Step 6:

Create two queue subscriptions, referencing the same Topic Name (EmployeeChanges) but reference the different queues (EmpReceiver1 and EmpReceiver2).

We will populate the topic EmployeeChanges in the first SAP Cloud Platform Integration iFlow which will then populate the queues EmpReceiver1 and EmpReceiver2



Step 7:

The Queue Subscriptions should look like this:. One topic with two queues that will be populated



Step 8:

Navigating back to the Queues, the Number of Messages in the queue is 0 for both which is what we expect



Step 9:

We'll now take a look at the Service Key (Step 3) that will provide us the authentication token and urls for the SAP Cloud Platform Integration development.



1) Refers to the details for the Management of the SAP Enterprise Messaging service. Eg create/update queues, stats, delete etc. API details here

2) Refers to the interacting with the Messaging of SAP Enterprise Messaging service. Eg Publish a message to a queue/topic, read a message off a queue/topic. API details here

3) Refers to the oauth token we'll use to authenticate from SAP Could Platform Integration to SAP Enterprise Messaging. clientid = username, clientsecret = password

4) Refers to the url we'll use in SAP Cloud Platform Integration to push/read messages

In SAP Cloud Platform Integration, deploy a security artifact (OAuth2 Credential)



Token Service URL = tokenendpoint (from the Service Key above)

Client ID = clientid (from the Service Key above)

Client Secret = clientsecret (from the Service Key above)

 

Step 10:

Create the SAP Cloud Platform Integration iflow to populate the Topic. We'll query SuccessFactors to get data and push it to the Topic. Of course this could be anything (eg from S/4, a third party system etc)

1) Call SuccessFactors to retrieve employee changes.

This can obviously change with parameters to eliminate hardcoding etc, but for now, just a simple request to get some data. NB: in real life, consider what you constitute as a message (eg one employee/event)

SELECT person, personal_information FROM CompoundEmployee WHERE last_modified_on >= to_datetime('2018-01-11T17:59:59Z')

2) Set the HTTP Header required for the API call

Content-Type = application/xml

x-qos = 1 (When we consume this message later, we want a confirmation message to confirm delivery and to remove it from the queue)

3) Post the message to the Topic

Address = From Step 9 the uri + messagingrest/v1/topics/<Topic Name>/messages (from the API documentation)

eg https://enterprise-messaging-pubsub.cfapps.eu10.hana.ondemand.com/messagingrest/v1/topics/EmployeeCh...

Authentication: OAuth2 Client Credentials

Credential Name: The credential deployed in Step 9

Step 11:

Deploy the interface!



Step 12:

Navigate back to SAP Cloud Platform, and the SAP Enterprise Messaging Administration and look at the Queues

Forgot? Go to your Trial Account - Go to your Sub Account - Go to your Space - Go to Services then Service Instances - then click the Dashboard Button



Presto! the one message from SuccessFactors has been pushed to queues EmpReceiver1 and EmpReceiver2 via the topic EmployeeChanges



Step 13:

Create a second iFlow to read a message from the queue EmpReceiver1 to an SFTP server and confirm the message off the queue. You can replace the SFTP server with something else that is available you may have available. eg datastore. Obviously add exception handling etc for a real interface

1) Set the HTTP header records required for the requests to SAP Enterprise Messaging

Content-Type = application/xml

x-qos = 1

2) Call SAP Enterprise Messaging to retrieve the message from the queue

Address: Use the URI from Step 9 followed by /messagingrest/v1/queues/<QueueName>/messages/consumption where queue name = EmpReceiver1 in this scenario

Credential Name: The credential deployed in Step 9

3) This step pushes the message to an SFTP server as a Send step.

Important: If you're using something else (eg SOAP), you can use a sequential multicast is used instead to ensure the message is delivered first, before confirming it off the queue. (eg if it delivery fails, then the message is not confirmed off the queue and hence can be picked up again)

4) Call SAP Enterprise Messaging to acknowledge the message as delivered and therefore can be removed from the queue

Address: Use the URI from Step 9 followed by

/messagingrest/v1/queues/<Queue Name>/messages/${header.X-Message-Id}/acknowledgement

where queue name = EmpReceiver1 in this scenario. The header value: X-Message-Id is the message id that is returned in the first query. This is detailed in the API documentation referenced previously

Credential Name: The credential deployed in Step 9

5) Add the X-Message-Id as an Allowed Header on the iFlow

Deploy the iFlow

Step 14:

Check the iFlow process



Step 15:

Check the destination. In this case an SFTP server



Step 16:

Check the Queues in SAP Enterprise Messaging

Forgot? Go to your Trial Account - Go to your Sub Account - Go to your Space - Go to Services then Service Instances - then click the Dashboard Button (Or follow Step 4)



The message has been consumed from the queue EmpReceiver1 and the queue is empty. EmpReceiver2 still has the message and can be consumed when/if needed.

Done!

Summary:

You've now published a message to a topic, had two queues populated with this data and consumed the data off one of the queues.

Why is this helpful?

When the next system comes with the requirement to get employee changes, you just need to create another Queue Subscription (steps 5 and 6), then create an interface that will consume the data off the queue rather than calling SuccessFactors again! Therefore no re-developing or copying code to call the sending system again for the data across integration developments. Consider pushing the whole data record and let each receiver determine pick and choose the data they want.

Additionally it gives flexibility for each receiver to consume the data as required (eg schedule the consuming iflow to poll every day, hour, week.)

In my opinion it clearly decouples the sender from receiver(s) while also enabling restart for each receiver independent from each other.

Hope this was helpful

Cheers,

Jon