Skip to Content
Technical Articles
Author's profile photo Dries Van Vaerenbergh

Cloud Foundry Enterprise Messaging Webhooks ?

Hi Community!

I would love to share this blog with you guys about “Cloud Foundry Enterprise Messaging Webhooks ?”. This because it is an “easy” solution to get real-time messaging events from wherever you want to wherever you desire. When I say real-time, I’m talking about Webhooks, web-sockets and so on, no polling and pulling!

This will be a Blog series with the following topics in separated Blog posts:

Cloud Foundry Enterprise Messaging Webhooks ? (this blog)
Send AMQP messages from CPI to Enterprise Messaging and Consume them in a Node.js AMQP Application ?
WebSocket’s in SAP UI5 Multi Target Applications consuming Enterprise Messaging ?

 

Introduction

First things first, this blog is not going to be about “What is Enterprise Messaging” or “How to get started with enterprise messaging”. To get started and go all the way I would really recommend having a look at the SAP Help Documentation:

https://help.sap.com/viewer/bf82e6b26456494cbdd197057c09979f/Cloud/en-US

It is the best way to setup, get started and make some advanced cool demo app/projects.

It is also the way I started to discover and get hands on with this Enterprise Messaging service in Cloud Foundry.

Now why this blog? Because I love real-time data in my applications, and I hate the pulling/polling implementations in applications. It is a bad practice and it can result in a lot of overheat and bad user experience. In short, your application will not be the best it can be.

It actually also makes a lot of sense to use Webhooks and or WebSocket’s. Let me clarify this “common sense” by the following example:

Imagine you are desperate for that one book in the library. But sadly enough someone else borrowed the book before you and did not yet return it. You do not want to go back to the library every single day over and over again. Cause this would be the pulling/polling effect. What you do want to achieve, is a scenario where the library staff members contact you once the book has been returned. This can be compared to the Webhooks/WebSocket’s implementation. Just “common sense” between the server and the client, like the customer and the library.

 

Webhooks <> WebSocket’s

Now before we start the setup of our Enterprise Messaging Webhook, it is important to understand the difference between Webhooks and WebSocket’s. I did some Googling myself and I found the following explanation on Stack Overflow.

Which gives you a good and nice idea on how these technologies work and how they differ from each other.

Webhooks

Webhooks are for server to server communication. They work by one server telling another server that it wants data sent to a certain URL when something happens.

WebSocket’s

WebSocket’s are (usually) for server to browser communication. The server hosts a WebSocket server, and clients can open a connection to that server. This is popular now mostly because it is faster and less resource-hogging than older ways of solving the problem like long-polling.

 

Setup the Enterprise Messaging Queue

To setup your Enterprise Messaging and Environment you can check out the following SAP Help Documentation:

https://help.sap.com/viewer/bf82e6b26456494cbdd197057c09979f/Cloud/en-US/3ef34ffcbbe94d3e8fff0f9ea2d5911d.html

Let’s say I have the following Queue inside my Enterprise Messaging Cockpit called “ErrorQueue”.

 

Send a Message to the Queue

To send messages to Queues you can check out the following SAP Help Documentation:

https://help.sap.com/viewer/bf82e6b26456494cbdd197057c09979f/Cloud/en-US/577ea7ce5cef4e2ea974c03d5549b3ff.html

Which will also point you to the API-Endpoint documentation:

https://help.sap.com/doc/3dfdf81b17b744ea921ce7ad464d1bd7/Cloud/en-US/messagingrest-api-spec.html

Let’s say I send the following message to my “ErrorQueue”:

{
    "errorMessage": "This error came from Postman!"
}

This HTTP-POST-Request will create the message above inside our “ErrorQueue” Queue. It will return the HTTP-status-code “204 No Content” when successful.

When you perform a GET request to retrieve all your Queues, you will get a result like this:

If you have a look at your “ErrorQueue” inside the Enterprise Messaging Cockpit, you will also see the number of messages inside the Queue.

 

Building a Node.js Webhook

If you have a look at the Webhook definition above, you see that you need a second server. Wait where is my first one? Well, this is the Enterprise Messaging Service itself. It is able to receive message/notifications via REST, MQTT and AMQP, which means it is also possible to send messages via those protocols to the Enterprise Messaging Queues and Topics. But there are some “Prerequisites and Restrictions” in receiving and sending messages, which can be found here:

https://help.sap.com/viewer/bf82e6b26456494cbdd197057c09979f/Cloud/en-US/ac83090b07684f8e908df40d024f8fe5.html

In the end it is no more than the Enterprise Messaging Server performing a POST-request on the URL you provided as Webhook.

This means we need to provide an Endpoint inside our Node.js Application which allows us to perform a POST request. We will use “Express” to build such a server and to expose our endpoints.

To build a Node.js Application and deploy it to Cloud Foundry you can follow the following SAP Help documentation:

https://help.sap.com/viewer/65de2977205c403bbc107264b8eccf4b/Cloud/en-US/772b45ce6c46492b908d4c985add932a.html

Inside the “index.js” file you add an endpoint called “/emMessages” and you make sure it is an endpoint which allows us to perform a POST-request on.

In the end your code will look like this:

const express = require('express');
const app = express();
const bodyParser = require('body-parser');

// support parsing of application/json type post data
app.use(bodyParser.json());

//support parsing of application/x-www-form-urlencoded post data
app.use(bodyParser.urlencoded({
    extended: true
}));

app.get('/', function (req, res) {
    res.send('Hello World!');
});

app.post('/emMessages', function (req, res) {
    console.log(JSON.stringify(req.body));
    res.status(201).send();
});

const port = process.env.PORT || 3000;
app.listen(port, function () {
    console.log('App listening on port ' + port);
});

As you can see the “body-parser” package is required and used on our “app” (express server). This allows us to read the payload from the POST-request.

Once we receive the data on this “/emMessages” endpoint, we log to the console.

With this you finished the development of your Webhook for the Enterprise Messaging. You deploy this application to your SAP Cloud Foundry Space with the following command:

cf push

Just like it is explained in the SAP Help documentation above.

With this you deployed your Webhook and you can configure it in the SAP Enterprise Messaging Cockpit.

 

Setup the EM Webhook Subscription

With the Node.js App deployed, which fulfills the role of a Webhook, we are ready to use this app inside the Enterprise Messaging Cockpit. But first the Webhook URL needs to be retrieved. Go to your SAP Cloud Foundry Space Applications and select your Webhook App.

Once you selected the Webhook app, you will see the URL.

Copy paste this URL since you will need it in the Enterprise Messaging Cockpit Webhook Subscriptions” configuration panel.

In the Enterprise Messaging Cockpit under the “Webhook Subscriptions” a new configuration needs to be created. This with the following configuration values:

Configuration table for the input fields above:

Config

Value

Subscription Name Free to choose
Queue Name Queue you want the Webhook the subscribe to. In this case “ErrorQueue”.
Quality of Service (QoS) 0 (No acknowledgement has to be sent after consumption this way)
Webhook URL The URL of you Webhook in CF. (You just copied it) It needs to start with https:// followed by the URL and it ends with the “/emMessages”. This because that is the endpoint which was implemented to receive the messages.
Exempt Handshake YES (we do not need the handshake acknowledgments between our Webhook app server and Enterprise Messaging)
Authentication NoAuthentication (Since no authentication method was provided inside the Node.js App, could be OAuth2 with passport and xsuaa)

Once you created the “Webhook Subscription” it will appear in the overview:

 

Examine messages in the Node.js Webhook

At the moment there is still one message in the “ErrorQueue”, because it was sent to the Queue before the Webhook was created. But Enterprise Messaging picked up this message and sends it over to the Webhook URL once created. Depending if your Webhook was already up and running, the message will be logged or not.

Resend the message with the following payload to the Enterprise Messaging “ErrorQueue”:

{
    "errorMessage": "This error came from Postman!"
}

Have a look at the Node.js Webhook Application in the SAP Cloud Foundry Space. Select the Application and choose the “Logs” section in the left-hand-side menu.

As you can see the message has been logged inside the Node.js application.

 

Wrap up ?

With this setup, configuration and development, you created a Node.js Webhook application for the Enterprise Messaging Service and you subscribed your Queue to it. A Webhook is no more then an endpoint which allows you to perform an HTTP-POST request on. This is exactly what the Enterprise Messaging does, once a Webhook Subscription has been configured. It receives messages and it “forwards” it to the Webhook endpoint.

In the next blog we will have a look at “Send AMQP messages from CPI to Enterprise Messaging and Consume them in a Node.js AMQP Application ?”. I hope you found this blog helpful and interesting and I’ll see you in the next blog!

Kind regards,

Dries

Assigned tags

      22 Comments
      You must be Logged on to comment or reply to a post.
      Author's profile photo Fatih Pense
      Fatih Pense

      Great series Dries! Thanks for sharing!

      Regards,
      Fatih

      Author's profile photo Dries Van Vaerenbergh
      Dries Van Vaerenbergh
      Blog Post Author

      Thank you Fatih!

      It's a pleasure!

      Kind regards,

      Dries

      Author's profile photo Helmut Tammen
      Helmut Tammen

      Thank you Dries. Great articles.

      Regards Helmut

      Author's profile photo Dries Van Vaerenbergh
      Dries Van Vaerenbergh
      Blog Post Author

      Thanks for the nice feedback Helmut!

      Kind regards,

      Dries

      Author's profile photo Mahesh Kumar Palavalli
      Mahesh Kumar Palavalli

      Nice one Dries Van Vaerenbergh !!

      Author's profile photo Dries Van Vaerenbergh
      Dries Van Vaerenbergh
      Blog Post Author

      Thanks Mahesh! I really enjoyed the Websockets implementation along with the Enterprise Messaging Service!

      Author's profile photo Carlos Roggan
      Carlos Roggan

      Hi Dries Van Vaerenbergh ,
      Honestly, since long time I wanted to understand what's about those hooks - until I found your blog which made me happy !
      😉
      Cheers,
      Carlos

      Author's profile photo Dries Van Vaerenbergh
      Dries Van Vaerenbergh
      Blog Post Author

      Realy appreciate it Carlos! Thanks a lot!

      Best regards,

      Dries

      Author's profile photo Varun Khetan
      Varun Khetan

      Hello Dries,

      is it possible to get X-Message-Id when receiving an event from Webhook?

      So that an acknowledgement can be provided.

      My requirement is to keep the message in Unacknowledged Queue, if the downstream application is down during maintenance.

      Thanks,

      Varun

      Author's profile photo Dries Van Vaerenbergh
      Dries Van Vaerenbergh
      Blog Post Author

      Hi Varun,

      Thanks for your good question!

      To be honest I had no idea, I had a look at it myself and at the moment I did not find an answer for your question.

      I do not see the X-Message-Id neither when I consume the messages via another Node.js app using @sap/xb-msg and not webhooks.

      Also when I create a webhook in the Enterprise Messaging Dashboard and i set the QOS to 1 while I send a message with x-qos to 1 the message is still consumed and no longer present in the queue.

      I would expect it to be still present in such a case since the acknowledgement is still to send, but it is totally consumed.

      What you could do, but I do not recommend and always try to avoid it is to create an application that polls the the messages via the rest API-endpoint and then sends the acknowledgement after reading the headers. But that just sounds very bad and I believe there should be another way to find out that  X-Message-Id, but I can't tell how for the moment.

      Good luck!

      Best regards,

      Dries

       

      Author's profile photo Deniz Cengiz
      Deniz Cengiz

      Hi Dries Van Vaerenbergh,

      first of all great post and great series 🙂

      If you don't mind me asking, why do you strongly recommend not to use the REST API to consume events (with QoS set to 1) and subsequently acknowledging consumption with the X-Message-ID header provided by the message by calling the acknowledgement-endpoint of SAP EM REST API?

      If network reliability is of high importance and webhook subscriptions seem to not properly cover the acknowledgement of message consumption by simply deleting messages from the queue, what's your way to go to achieve a behaviour that adheres to a QoS=1 / ATLEAST_ONCE?

      Thanks in advance!

      Best regards,

      Dan

      Author's profile photo Tobias Griebe
      Tobias Griebe

      Hi Deniz Cengiz,

      and this should also help with Varun Khetan's question:

      When creating the WebHook you can set the QoS = 1. That means whenever a message arrives in a queue it's pushed via the WebHook functionality to the configured REST endpoint.

      A message is acknowledged only after the webhook returns a 2xx response.

      If a webhook does not return a response within 1 minute, or if the webhook returns a non-2xx status code, the message is redelivered to the webhook after 15 minutes. The retry attempts continue until the webhook acknowledges the message with a 2xx response. Otherwise the message expires on reaching its expiry time (a.k.a. TTL or time-to-live).

      A webhook can only hold on to a maximum of 9 unacknowledged messages, beyond which no new messages are sent to the webhook.

      Messages consumed with webhook subscriptions cannot be acknowledged using the Acknowledgement API (POST /queues/{queue-name}/messages/{message-id}/acknowledgement).

      If a webhook subscription is paused or deleted, all unacknowledged messages held by this webhook are released to the queue and made available for consumption.

      Best regards,
      Tobias

      Author's profile photo Deniz Cengiz
      Deniz Cengiz

      Hi Tobias Griebe

      thanks for your quick answer and your input. I was aware that you cannot use the acknowledgement API when using webhook subscriptions, my question was more targeted towards still guaranteeing message delivery, as both Varun and Dries seemed to have issues with acknowledgement.

      Couple of quick questions for you, if you don’t mind.

      • Those limitations and facts you mentioned (retry for 1 minute for non 2xx reponse; 9 unack messages per hook, etc) – are they documented somewhere you could link me to?
      • Also do I understand you correctly: Whenever a webhook with unack messages is paused (or deleted), these unack. messages (up to 9 according to your comment) will go back to the queue and be persisted once again?
      • Going further, does that mean that for as long as a message is unacknowledged, it is held via the webhook and kind of “lost” inbetween (while retry is happening) until it’s either consumed, its TTL exceeds or the webhook subscription is paused/deleted? What I mean is that it seems to be deleted from the queue (going with Dries comment here), but it is actually being held by the hook, thus not traceable?
      • What happens in case 9+ message get unacknowledged? Does the EM queue automatically stop publishing messages to the webhook/endpoint? Is the endpoint automatically paused?

      Thanks in advance.

      Best regards,

      Dan

       

      Author's profile photo Tobias Griebe
      Tobias Griebe

      Hi Deniz Cengiz,

      the numbers that I have listed are not officially documented and can change at any time without further information. Still it is what you can expect how it will behave for now.

      Whenever a webhook with unack messages is paused (or deleted), these unack. messages (up to 9 according to your comment) will go back to the queue and be persisted once again? That's correct. It might take a little time until this is completed, but no messages are getting lost.

      When using QoS=1, the message is NOT deleted from the queue unless it is acknowledged from the consumer. You can check in the administration UI: for your queue you will see the number of unackowledged messages.

      If you have more than the 9 unackowledged messages, the webhook will stop pushing new/additional messages to the endpoint. It will continue to retry for those 9 messages. As soon as there are less than 9 unackowledged messages it will continue to push new messages. But this doesn't pause the webhook itself.

      Best regards,
      Tobias

      Author's profile photo Sandesh Kurumella
      Sandesh Kurumella

      Hi Tobias,

      Is there a reason why the webhook will stop pushing new/additional messages to the endpoint if the unack messages are greater than 9 ?

      Can we not still push new messages irrespective of the count of unack messages?

      The reason asking these questions is, we are implementing event messaging at one of our customers and we have a scenario where we still want to push additional messages irrespective of the unack messages.

      Regards,
      Sandesh

      Author's profile photo Tobias Griebe
      Tobias Griebe

      Hi Sandesh,

      the number of supported unacknowledged messages is defined by the implementation of the webhook. Currently there is no way to configure this number by the end user.

      So once there are 9 unacknowledged messages, no additional messages will be sent.

      It would make sense to understand the use-case in detail: Does the message consumer take that much time for processing the messages, or why would you need to have a higher amount of unacknowledged messages?

      If the consumer would be able to subscribe to EM via MQTT or AMQP, you could also have a higher number of unackowledged messages. The 9 messages is just a limitation of the webhook functionality.

      But we are working on a concept to enhance the webhook experience and currently we are also considering to make the number of unacknowledged messages configureable. Right now this is still in concept phase, so not sure if this will be available and we don't have any timeframe for it.

      Regards,
      Tobias

      Author's profile photo Mikael Gurenius
      Mikael Gurenius

      Hi Tobias! Is there a way to configure the retry attempts for a webhook? I'd like an exponential retry policy that doubles with each try attempt.

      The property 'deadMsgQueue' is available for a queue but I can't seem to find a way to configure that either. Is there a way to let a webhook "give up" after N number of attempts and forward to a Dead Message Queue?

      Best regards,

      Mikael

      Author's profile photo Tobias Griebe
      Tobias Griebe

      Hi Mikael,

      at the moment there is no option for you to configure the retry policy.

      Last time I did some test for the retry intervals I got the following times (but it might have changed in the meanwhile):

      • Once in 15 mins. for 12 hrs.
      • Once in 30 mins. for next 12 hrs.
      • Once in 60 mins. for next 24 hrs.
      • Post that once a day for 28 days.

      So you see, the intervals are already increasing.

      Using the management APIs you have seen the attribute for 'deadMsgQueue'? So far there is no documentation on that feature available. As far as I know it's unfortunatley not bound to the retry mechanism, but only to the TTL of a message. If a queue is configured with DMQ, expired messages end up in the DMQ instead of getting lost.

      But we are working on a concept to enhance the webhook experience. At present it is not possible to configure the retry strategy, but then we are evaluating a few alternatives to give customers more control. Right now this is still in concept phase, so not sure if this will ever be available and we don't have any timeframe for it.

      Regards,
      Tobias

      Author's profile photo Mikael Gurenius
      Mikael Gurenius

      Thanks! I'll continue exploring...

      Author's profile photo Khurram Mir
      Khurram Mir

      Dries Van Vaerenbergh Thanks for sharing such informative article. I am not able to access  YouTube playlist mentioned  at the start of the blog. Also, i am not able to access the YouTube videos embedded into this blog. I am just wondering if you have removed these videos or it is something related to my account only? Can you please share updated URL?

       

       

      YouTube%20Video%20Error

      YouTube Video Error

       

      YouTube%20Playlist

      YouTube Playlist

       

      Author's profile photo Dries Van Vaerenbergh
      Dries Van Vaerenbergh
      Blog Post Author

      Hi Khurram Mir thanks for your feedback and to inform me about the videos.

      I just checked it out myself and the videos are put privately it seems or they have been removed.

      The videos are not mine, I only referred to the SAP HANA Academy their tutorial videos since they were really informative and nice to get started with Enterprise Messaging.

      I will try to update the blogs by replacing the videos with the SAP Help documentation.

      Once done, I will tag you in these comments again.

      In the meanwhile you could have a look at the following live stream replay where we also send messages to a queue:

      https://www.youtube.com/watch?v=DKI2WgDS_k0

      Thanks for the update!

      Best regards,

      Dries

      Author's profile photo Dries Van Vaerenbergh
      Dries Van Vaerenbergh
      Blog Post Author

      I updated the blogs with the SAP Help Documentation links.

      Enjoy!