Update 29 Oct 2014: Updated Resolution section - for ICO scenarios, processing is done on the worker threads on the sender side.
Background
Since the early days of XI, getting messages stuck in the To Be Delivered status on the Adapter Engine has been a recurring issue. A search on SCN will list down hundreds of such discussion threads.
OSS note 813993, has an FAQ with the details of the different status in the Adapter Engine. It states that TBDL is a temporary state and cannot be changed manually. Therefore, it is not possible to Resend a message in this state from RWB.
To Be Delivered (German: "zu senden", internal: TO_BE_DELIVERED, TBDL)
The message was transferred to the messaging system and was placed in the queue for the messages to be sent or received. The status is temporary and cannot be changed manually. Generally, this is not necessary, since the message is taken from the queue by a worker thread. For a large queue, you can start such a message manually in order to raise its priority.
Analysis and Troubleshooting
To troubleshoot further, first search for all messages in status "To Be Delivered" in the Adapter Engine. Change the time/date criteria to go as far back to identify when blockage first occurred. Then search for messages in status "Delivering" around the timeframe of when the block first occurred.
Once you find these, refresh the display a few times, and if the messages do not change status, then it is most likely that they are the messages causing the blockage.
You can then check the audit log for any further details of why the messages are still in "Delivering" state. If there are no details in the audit log, it could be because they are already flushed from the cache (refer OSS note 1314974.)
You can also further verify that the AE queues are blocked by the checking the engine status.
RWB -> Component Monitoring -> Adapter Engine -> Engine Status -> Additional Data
Over here, a large value in "Number of Entries in Queue" will indicate the blockage. You will also notice that the Assigned/Working threads are equal to the Max Threads. If you click into the queue, you would be able to see the messages in the queue.
Resolution
When you have found the cause of the blockage, you need to resolve the issue before resending the blocked messages.
On newer versions or SP level of XI/PI, it is possible to release the threads held up by the "Delivering" messages by stopping the corresponding channel. This will require the correct version/SP level as described by OSS note 1604091.
For classical scenarios, normally the hanging threads are at the receiver channel. However, for ICO-based scenarios, the hanging threads will be at the sender channel as the messages are processed from the send queue (refer to Mike's blog below for further details.)
As this issue often happens on the receiver end, one of the common cause of this issue is due to connectivity problem with the receiving backend system. This might be due to an FTP server not available or a JDBC server not having enough resources. After the backend issue has been resolved, firstly the blocking messages should be resend. After that, the "To Be Delivered" messages should automatically start to clear.
However, on older releases, the message status could not be updated correctly and this might require a restart of the Java stack as mentioned in many threads.
Prevention
While a Java stack restart might resolve this issue after it has occurred, that does not prevent it from happening again.
The common reason for the blockage is because the worker threads have been fully used up, or not released properly. This occurs quite frequently on FTP and JDBC receiver adapters, but also possible on other adapters. The following OSS notes provide more details:-
849089 - sporadic connectivity to the FTP server can cause the channel to hang indefinitely
1473299 - setting of poolwaiting and concurrency causes adapter to wait indefinitely
1136790 - Blocking receiver channel may affect the whole adapter type
The following can be done to prevent the adapters from hanging.
1) Set timeout and connect mode in FTP adapters (for all FTP sender and receiver channels)
2) Set a non-zero pool waiting time in JDBC channel
3) Optionally set the queueParallelism properties (following note 1136790 for classical scenarios and note 1493502 for ICO scenarios) to prevent any single interface blocking the whole adapter
NWA -> Configuration Management -> Infrastructure -> Java System Properties -> Details -> Services
This timeout setting can also be extended to other adapters as necessary depending on the availability of such setting in the adapter.
References
Mike's blog on tuning the PI message system describes in detail more about the worker threads, and also more details for further tuning.
OSS note 1623356 - "To be delivered" messages in Adapter Engine
- SAP Managed Tags:
13 Comments
