Skip to Content

This blog describes options for configuring asynchronous one-way message processing using the new JMS adapter, which is now available for customers. It describes the setup, configuration, size limits and monitoring of your scenario. The JMS adapter is available for SAP Cloud Platform Integration customers with an Enterprise Edition license since 24-June-2017 release (2.29*).

Asynchronous Messaging with Retry Using the JMS Adapter

In many cases integration scenarios have to be decoupled asynchronously between sender and receiver message processing to ensure that a retry is done from the integration system rather than the sender system. This can be achieved, for example, by using JMS queues to temporarily store the messages in the cloud integration system if the receiver system cannot be reached, and retry them from there. Follow the steps described below to activate the message broker and to setup the scenario. Use the monitoring tools described to check the processing and view the stored messages in the queue monitor.

Prerequisite: Broker Provisioning

To be able to use JMS queues for storing messages you first need to activate the message broker for your cloud integration tenant. If you have an Enterprise Edition license you can do this for your tenant in Cloud Integration Provisioning via Provision Broker. You can see then the active message broker and its details. You can now start using the JMS adapter to configure your scenario.

This is described in detail in blog ‘How to activate Message Broker’.

Setup Scenario With Asynchronous Decoupling

To configure the decoupling of inbound and outbound message processing you need to configure two processes:, one process to receive the inbound message and store it in the JMS queue and a second process to trigger the message from the JMS queue to the receiver backend. The blog describes the configuration using two separate integration flows.

Configure the Integration Flow Receiving the Message

The first integration flow will be configured to receive the message via any inbound adapter. In this sample setup we use the HTTP adapter.

Configure the JMS Receiver Channel

Create the integration flow with the inbound channel required by your scenario, and use the JMS adapter as the outbound adapter. You only have to configure the Queue Name for the JMS queue that will be used to store the messages. It is Important to set an Expiration Period that suits your scenario. The default value is 90 days’, which means that after 90 days any messages, that were not delivered within this period will be deleted from the queue.

With the 10-December 2017 release, you will have the option to configure, if exchange properties are forwarded to the JMS queue. Before, the properties were always forwarded along with the message, which caused severe issues in some scenarios. The default value of the option will be that properties are not forwarded, because this could lead to runtime issues if the properties are too large. In detail the size restrictions are described in the Blog ‘JMS Resource and Size Limits’.

This option is available in adapter version 1.1.

Configure Transaction Handling

You need to configure the correct transaction manager in the integration process for transactional end-to-end processing. Our process does not contain any JDBC resources (data store, variables, aggregator), and only one direct JMS Receiver without splitter or sequential multicast; so we don’t need a specific transaction handler. Select the integration process and switch to the Processing tab. In Transaction Handling drop down select Not Required.

More details about the different transaction handling options and existing limitations are described in the blog ‘How to Configure Transaction Handling in Integration Flow’.

Deploy the Integration Flow

Now you can deploy the integration flow. During deployment, the specified queue will be created automatically in the message broker. You can check that the queue has been created in the Queue Monitor available in the operations view in the Manage Storages section. The monitor is described in more detail in the Monitoring section below.

If a message broker has not yet been provisioned, the deployment will end with an error. In Manage Integration Content the error details for your integration flow tell you that no messaging host is available.

Configure the Integration Flow Doing the Retry

To consume the messages from the JMS queue, you configure a second integration flow with a JMS sender channel and the outbound adapter needed for your scenario. In this sample configuration we use the HTTP adapter.

Configure the JMS Sender Channel

Create the integration flow with the outbound channel required by your scenario, and use the JMS adapter as the inbound adapter. You only have to configure the Queue Name for the JMS queue that you want to poll the messages from. Use the same queue name used in the receiving integration flow.

Number of Concurrent Processes is set to 1 per default and should only be increased if more parallel processing is needed. Because the JMS resources are limited, you need to keep the number of parallel processes as low as possible. Especially if large messages are processed in the scenario, increasing the parallelism may lead to out of memory problems.

In the JMS sender channel you also have to configure the retry handling. Set the Retry Interval as required for your scenario (, the default value is 1 minute). We also recommend using Exponential Backoff, which means that the retry interval is doubled after each unsuccessful retry. By using this setting you can avoid calling the receiver backend every minute if it is unavailable for a longer time period. The Maximum Retry Interval defines the maximum time between two retries if exponential backoff is used. The recommendation is to either keep the 60 minutes or even increase it if this is acceptable by the scenario.

Note, that the JMS sender does not guarantee the order in which the messages are consumed. If several processes are configured to consume messages and/or multiple worker nodes process messages, the messages are processed in parallel. Furthermore, in case the message is in retry because of an error it is taken out from processing until the retry interval is reached.

With 22-July-2017 release the option to configure a dead letter queue is introduced. This is described in detail in blog How to configure Dead Letter Handling in JMS Adapter.

Configure Transaction Handling

If you are using JMS Sender, it is not necessary to set the transaction handling to Required for JMS, because the retry handling works independently of the selected transaction handler. You should select the transaction manager as required by the scenario that you have configured in the integration flow.

In the simple example shown, no JDBC or JMS resources are used, that need a transaction handler, so we select Not Required.

There are some limitations on configuring transaction handling configuration and on which flow steps can be used in which combinations. This is described in detail in the blog ‘How to configure Transaction Handling in Integration Flow‘.

Optional: Explicit Retry Configuration

In the JMS sender channel, you can configure the time intervals between retries, but you cannot configure that processing will end after a specific number of retries. If required, you can configure this in an exception subprocess that calls a local process for retry handling.

To configure this, add a Local Integration Process to the integration flow. In this process you configure the specific retry handling. In this sample, we check the number of retries executed and after 6 unsuccessful retries, we end the processing and trigger a mail to alert someone to the problem.

In the local integration process configure a Router after the start event, add an Error End Event and configure the router as shown in the picture. The header SAPJMSRetries set by the JMS adapter is used to decide if the message processing continues or if it ends and a mail is triggered. If message processing is to continue, an error is raised by the process so that the message stays in the JMS queue and goes into retry status.

We set the transaction handling to From Calling Process in the local process, and as there are neither JMS nor JDBC transacted resources configured in the local process, we take over the Not Required setting from main process.

Furthermore, you need to add an Exception Subprocess in the main process, and add a Local Process Call, where you select the configured local process, within the exception subprocess. Add a Receiver for the mail to be sent. Connect the Message End Event of the exception subprocess to the receiver using a Mail adapter receiver channel. In this channel, you configure the mail address the mail is to be sent to and the details to be sent.

With this configuration the message will be removed from the JMS queue after 5 unsuccessful retries, so that no further retries are executed. A mail is triggered to notify the appropriate person to take manual action.

If you use the explicit retry configuration, you are free to configure retries as required for your scenario. For example, you can notify someone after 3 retries but not stop the processing until 10 retries have been made, to allow enough time for manual actions. Or, you can send the message after 3 retries via an alternative receiver channel.

Deploy the Integration Flow

You can now deploy the integration flow. This completes the scenario configuration in SAP Cloud Platform Integration and you can start processing messages. Test the configuration by sending a test message via the configured inbound adapter and check that it is received by the receiver backend.

JMS Resource and Size Limits

The connected JMS messaging instance, that is used in the asynchronous messaging scenario has limited resources. The Enterprise License sets a limit on the queues, storage and connections that you can use in the messaging instance. The limits are described in more detail in the blog ‘JMS Resource and Size Limits‘.

Monitoring

To check the processing in detail, you can use the monitoring tools provided by Cloud Integration.

Integration Flow Monitoring

In the operations view, in the section Manage Integration Content, you can find the deployed integration flows and their status. Check that the flows have the status Started.

Message Monitoring

Message processing can be checked in detail in the section Monitor Message Processing. Select the message you sent and check the processing status. There should be two message processing logs, one for the inbound messaging that saves the message to the JMS queue, and one for processing the message from the queue and sending it to the receiver system.

Those two messages are correlated via a Correlation ID. With the 8th-July-2018 update of Cloud Integration, this Correlation ID is shown in the Properties of a message processing log. You can also search for all messages having the same Correlation ID using the ID search field at the top of the monitor:

In the event of a failure when sending to the receiver system the outbound message will go into status Retry. The message will then be retried as configured in the JMS sender channel. The detailed error information and the retries are available under the link Message Processing Log.

Queue Monitor

If there are problems during processing and your message does not arrive at the receiver backend, the message status is Retry and the messages can be checked in the Queue Monitor in the section Manage Storages in the Operations view. This monitor shows all queues existing in your tenant, with their stored messages. Select your queue to check the messages contained in the queue.

You can see the number of retries that have already been executed and the time of the next retry. You can also trigger an immediate retry of the message or delete specific messages if you no longer want them to be processed.

If a message broker has not yet been provisioned, the Queue Monitor tile is not shown.

Queue Deletion

If you do no longer need a specific queue, (for example if you un-deploy the integration flow), you have to delete the queue manually in the Queue Monitor. Unused queues, that do not contain messages will be deleted automatically after the May-13-2018 update. Queues containing messages cannot be deleted automatically because this could cause inconsistencies, as the messages in the queue may still have to be processed.

To find unused queues in the monitor you may use the checks described in the blog ‘Checks in JMS Message Queue Monitor’.

To report this post you need to login first.

15 Comments

You must be Logged on to comment or reply to a post.

  1. Vijayashankar Konam

    Thanks  for the detailed blog Mandy . This had been a long pending feature on SCI. However, hope SAP implemented the EO and EOIO QoS out of the box rather than customers to develop their own scenarios.

    (0) 
  2. Beverely Parks

    Thank you for this blog.  It is very helpful.  I am just starting to attempt to use JMS queues.  How do you recommend setting up the JMS queues?  One per message type or one per Sender?  We receive multiple transaction types from a Sender with an end point for each message type.  Would you recommend creating a queue for each message type/end point or for the Sender?  If at the Sender, how would the actual processing IFLOW look?

    These messages are being sent to our On-Premise ECC system.  I’m looking into JMS queues as a means of reprocessing transactions that we receive while our On-Premise ECC system is down for maintenance.  

    (0) 
    1. Mandy Krimmel
      Post author

       

      Hello Beverely,

      in general both is possible, you could route the messages based on some routing condition (based on payload) into different queues, or put all in one queue directly after receiving and do routing based on message type later. I would recommend second, because so you save the messages directly after receiving and have the retry point there. Also, you only have a limited number of queues available, see https://blogs.sap.com/2017/10/04/cloud-integration-jms-resource-and-size-limits-in-cpi-enterprise-edition/ .

      BR,

      Mandy

      (1) 
  3. Beverely Parks

    Thank you for your response.  Please understand that I am new to Cloud Integration with no PI experience.  I am stuck on two points.

    #1 – We receive multiple transaction types from a 3rd party.  Let’s say Orders, Shipments, and Receipts.  We currently have one IFLOW for each transaction.  Upon receiving each, we assign the SAP application ID, save a copy of the payload, perform mapping and send the transaction to our On-Premise ECC system.  So IFLOW 1 sends to ECC address#1, IFLOW 2 sends to ECC address#2, etc.  Each IFLOW also contains an Exception sub-process to send an email notification in case of failure.   Is it best to send the data to the JMS queue before any of the current IFLOW processes, or to send the data to the JMS queue rather than directly to ECC On-Premise?

    #2 If I use just one JMS Queue for all 3 transactions, how do I read the transaction name (ns1) to determine the routing?

     

    Thank you for your time and any assistance that you are able to provide.

    (0) 
    1. Mandy Krimmel
      Post author

       

      Hello Beverely,

      as already said, both is possible:

      1. use the existing flows and just do async decoupling in each of them
      2. create one receiving flow with a JMS receiver channel and put all messages in one JMS queue and have a second flow polling the messages out, routing the messages based on a routing condition (e.g. via XPath in router flow step – more info can be found in online help) to the different receivers

      Advantages for first option:
      – you can keep the existing flows and just introduce decoupling in all the flows
      – flows/scenarios are independent
      — flows can be updated independently
      — higher throughput possible

      Advantages for second option:
      – one central processing flow, may be easier to maintain in futur
      – only one queue, may be an aspect if you need several scenarios with JMS, as only 28 queues are allowed in enterprise edition

      Best regards,
      Mandy

      (0) 
    1. Mandy Krimmel
      Post author

       

      Hi,

      to get details about the mpl id you can use the header: SAP_MessageProcessingLogID (${header.SAP_MessageProcessingLogID}).

      BR,

      Mandy

      (0) 
  4. Beverely Parks

    Good morning, Mandy

    From my tests, it appears that the value in  SAP_MessageProcessingLogID is the same as the value in the MessageGUID, which I am including in the email error notification within my JMS IFLOW

    MessageGuid         = AFqeyBdhY5MyWg1p2NPhanK5eVtS

     

    I would like to also include the ID for the originating message (the IFLOW that used a JMS queue as the receiver) so that the receiver of the email error notification can easily find the originating message:

    LocalPredecessors [
    AFqeyBfZqT_cWr2AhpDR8CVefpQz
    ]

    Is this possible?

    Thank you!

     

     

    (0) 
    1. Mandy Krimmel
      Post author

       

      Hi, ok got it wrong.

      There is no simple way to get the ID of the first message. What you can get is the Correlation ID that correlates the inbound and outbound message. With header SAP_MplCorrelationId you can get this ID. Using this ID you can then search in message monitor to get all messages belonging to this correlation ID. Note that this search is not possible in UI yet (only via the monitoring APIs), but will come in UI in one of the next takt updates.

      BR,

      Mandy

      (0) 
      1. Mandy Krimmel
        Post author

         

        Hello,

        the search via correlation ID will be available with the next update. Details see above in the monitoring chapter.

        Best regards,

        Mandy

        (0) 
  5. Beverely Parks

    Good Morning.   Is it possible to configure the JMS Queues so that the retry option is FIFO?  It would be good if there is a way to control the order that the transactions are retried to insure that they post into ECC in the order that they were originally received.

     

    Thank you!

    (0) 
  6. Rajendra Samal

    Hi Mandy,

    I have configured a similar scenario as explained by you in the above blog. The Inbound message is in json format and the message is successfully coming to CPI but in the outbound configuration it is failing and going to retry. The error is as below:

    Error Details: java.lang.IllegalStateException: Exchange property SAP_AuthHeaderName must not be null

    I have also tried by using a content modifier in the outbound scenario and specified Action: Create, Name: ACCEPT, Type: Constant & Value: application/json

    (0) 
    1. Mandy Krimmel
      Post author

       

      Hello,

      this error is thrown by the receiver HTTP adapter. I’m not really sure what you are trying to configure. Are you using principal propagation in the HTTP receiver? This requires the specific property SAP_AuthHeaderName. The property is set by the sender adapter. If you want to store the properties along with the message in the JMS queue you need to select the flag ‘Transfer Exchange Properties’ in the JMS receiver channel.

      But using principal propagation with an asynchronous decoupling is not really the best to do because the PP token is valid only for a short time interval (usually 30 minutes). This means if the message goes into retry for whatever reason the retries will never be successful after 30 minutes.

      BR,

      Mandy

      (0) 

Leave a Reply