[PUSH-PULL Scenario on CPI] Storing webhook data on SAP Cloud Platform Integration and making it available through a simple HTTP API

[mattisebastian]

Hi,

in this blog post, I want to describe how I built a temporary data store on CPI that stores Webhook data and makes it available through a very simple API.
Basically, I built a RESTful HTTP API that supports HTTP POST, GET and DELETE.

Our use case was simple: a cloud application writes data through a webhook which then needs to be written to a database.

My job was to create an endpoint to send data to and fetch it from. I will explain how I used the Data Store on SAP CPI to accomplish this task. Fetching the data and writing to the database was done with SAP Data Intelligence and will not be covered in this blog.

Overview

On SAP Cloud Platform Integration it is possible to create a plain HTTP endpoint that will be appended to the Tenant URL for example:

https://XXXX-iflmap.XXX.eu1.hana.ondemand.com/http/MY_ENDPOINT

Now you could send HTTP requests to that endoint and handle them in an Integration Flow on that tenant. In that Flow you can react to the different HTTP verbs by reading a variable called CamelHttpMethod which is created by the framework.

The router then looks like this:

Click the image to enlarge it. Here is the snippet:

${header.CamelHttpMethod} = 'DELETE'

Works analogous for the other HTTP verbs.

GET - what data is there?

The meat of this part is a Data Store Select Operation. It just takes a number of polled messages and will return that many messages from the Data Store. These messages are retrieved in XML format and stored in the Message Body like this:

<messages>

  <message id="06307923-e965-4b90-b1b8-72f81181534d">

    <YourContent>

      ...

    </YourContent>

  </message>

</messages>

Note the <YourContent> element that will enclose each message. I will explain how it is created in the next section.

That's it! In my case I also have to change the format a little bit, add paging and transform it to JSON but that's stuff for another blog 🙂

POST - take my data!

In order to GET messages, first, we have to receive some. That is done in the POST part of the Flow.

A Data Store Write Operation handles the persistence of a message. As I except any JSON as Input from the Webhook I have to transform it to XML before storing it. That is done in a JSON-to XML-Converter standard component which will also add the JSON root element (YourContent) you saw in the section above.

The settings for the JSON-to-XML converter

That's it for storing messages. Pretty straight forward.

DELETE - remove this entry!

Delete was the trickiest part and why I decided to write this blog post. I made a pretty mean discovery which gave me a few headaches and which I wanted to share with you.

The delete could have been super simple: the Data Store Select (You remember, in the GET part of this blog) has the option 'to delete entries after selection'. On the first glance, it looks perfect for temporary storage: you select a bunch of entries and delete them after they were fetched by the client that called the GET method.

The problem is: they are deleted in every case, even when there is a network error and they are not correctly transmitted to the client. So there is potential for data loss.

The way to fix it is to create a dedicated DELETE endpoint which takes a message id and forward it to the Data Store Delete Operation.

A normal delete looks like this:

DELETE XXX.eu1.hana.ondemand.com/http/YOUR_ENDPOINT/{id}

But how can I access {id} to tell the Data Store Delete Operation which entry to delete?

I found this information in the very nice blog by Dominic Beckbauer.

There are two steps to this:

1. In the inbound HTTP Adapter change your Address to /YOUR_ENDPOINT/*


2. Now you can access everything after the second / in your Iflow in a header called: CamelHttpPath

I stored the Id in an Exchange Property (IDtoDelete) by the following expression:

${headers.CamelHttpPath}

Exchange properties are handy key-value mappings, which are attached to the message and don't leave the Integration Flow like headers.

The Content Modifier looks like this:

Creating an Exchange Property by reading a header variable

 

The trouble with the CamelHttpPath

Due to a variable in Postman, my DELETE request looked like this (note the double slash '//' before http)

XXX.eu1.hana.ondemand.com//http/YOUR_ENDPOINT/050b730b-811d-4ed2-8569-bcb87a72a502

Which led to a strang content of CamelHttpPath: "NT/050b730b-811d-4ed2-8569-bcb87a72a502"

Essentially not only the id was captured but parts of the URL, which should definitely not happen. It was resolved after removing the second slash '/'.

 

Summary

In this blog, I showed how I created a RESTful API consisting of GET, POST and DELETE on SAP Cloud Platform Integration.

I explained the DELETE operation in detail and showed a strange bug and the solution.

 

Thanks for reading!

If you enjoyed this blog let me know in the comments and I will create a follow up describing

• How I counted how many entries are there in total and how many were fetched (poor mans pagination)


• How I checked if an entry exists before trying to delete it and added custom response codes

 

Also let me know if you have any questions, comments, improvements and so on. I am always glad to improve my writing and CPI game.

 

Best regards

Matti