This blog describes the Dead Letter option in JMS Sender adapter, which will be available for customers with the next the 22-July-2017 release (2.30*). It describes the feature, when to use is and monitoring options. The Dead Letter Handling in JMS Sender Adapter will be available for SAP Cloud Platform Integration customers with an Enterprise Edition license with the 22-July-2017 release (2.30*).
Dead Letter Handling in JMS Sender Adapter
There may be messages, most likely large messages, received in the cloud integration system, that have the potential to lead to an out of memory error in the worker node, because the processing of the message needs too much memory.
In case such messages are processed in an asynchronously decoupled scenario using JMS, the message is stored in the JMS queue via the inbound flow, where normally no memory consuming conversions are done. But in the outbound flow the message may lead to may lead to the described problem: The node fails due the lack of memory, Cloud Platform restarts the node and therefore the Integration Flow leading again to an out of memory error since the message will be polled again from the JMS queue. The node restart would try to process the message again and again, leading to complete node unavailability.
To avoid this situation, messages, where the processing stopped unexpectedly, are retried only twice. Afterwards the message is taken out from processing and stored in a dead letter queue. Manual action is necessary to restart or delete such messages.
Prerequisite: Broker Provisioning and Setup of JMS Scenario
To setup asynchronous scenarios using JMS adapter is described in detail in Blog ‘Configure asynchronous Messaging with retry using JMS Adapter’.
Configure Dead Letter Handling in JMS Sender Channel
In the JMS sender adapter the dead letter queue can be configured starting adapter version 1.1. The flag for switching on the Dead Letter Queue is available in tab Connection. Per default the flag is selected, meaning the dead letter handling is active. In case you are sure in your scenario only small messages are processed and no memory issues will occur you may deselect the flag to improve the performance, but keep the risk of an outage in mind.
In case you open an old integration flow the setting might not be available, because the version of the adapter is too old. The version can be checked using the Technical Information icon on the channel. In order to use the Dead Letter Handling option, delete the sender channel and add it new from the palette. You will then get the latest version.
To check, if there are messages moved to the dead letter queue and taken out from processing, you can use the monitoring tools provided by Cloud Integration.
Monitor Message Locks in Manage Lock Monitor
As soon as there is a message processed by the JMS sender adapter a processing lock for this specific message can be found in the Message Locks Monitor. The monitor can be found in the operations view, in the section Manage Locks. The messages belonging to the JMS sender adapter can be identified by the Component JMS. Source column gives the information about the JMS queue, the message is processed in. The entry has the pattern JMS:<queue name>.
Normally this entry is removed as soon as the processing of the message is completed. In case of a node outage the processing cannot be completed and during startup of the node marked as erroneous and the entry will disappear from lock monitor. After approximately 7 minutes a retry is executed. A second retry is triggered in case the processing could again not be completed because of another node outage.
After the second retry the message is moved into a dead letter queue, which can be identified by the component JmsDeadLetter. The messages belonging to this lock will not be processed anymore. Manual interaction is needed. Note down the queue name shown in column Source and the JMS Message ID shown in column Entry and search for them in Queue Monitor as described in the next chapter.
The Expires At timestamp correlates to the deletion timestamp of the JMS message configured in the JMS receiver channel.
Monitor Corresponding Message in Message Queue Monitor
To check the message that was taken out from processing go to the Message Queue Monitor, which can be found in the operations view, in the section Manage Stores.
Search for the queue you noted down in last step and show the messages contained in the queue. Using browser search for the JMS ID from Message Locks Monitor.
Using the direct link for the Message ID, you can directly jump to the message processing log for the message send into the queue. There you can find the integration flow name, the time the message was sent and the details of the processing.
To analyze, if the message was causing the outages you may download the message and check its size. After the analysis you need to decide, either completely remove the message from processing, change your integration flow and/or try again to resend it:
- If you decide to remove the message completely from processing delete the message in the Message Queue monitor and afterwards also delete the lock entry in Message Locks monitor. Ask the sender of the message to send it in smaller chunks, if possible.
- If you decide to retry the message, just delete the lock entry in Message Locks monitor. The JMS sender adapter will then trigger another retry. Carefully monitor, if the node crashes again. In that case you can be quite sure that it is the message, that is causing the outage.