Outbound IDoc: Status ’03’. Absolute Success? Not Necessarily!
In outbound IDoc scenarios (mediated via SAP PI/PO or used in point-to-point integration), it is not unusual to witness situation when support team or application consultant checks generated IDoc in a sender ABAP system (for example, SAP ERP) using standard built-in IDoc monitoring tools (like transactions WE02, WE05 or BD87), examines IDoc’s status record verifying that its last status is ’03’ (“Data passed to port OK”) and provides a notice of the IDoc being successfully sent by a system. It can also be a case in such situation, that a receiver system hasn’t received any corresponding inbound IDoc. As a result, it is commonly stated that the IDoc is lost or, if SAP PI/PO is used as a middleware system in the scenario, it is claimed that processing in PI/PO failed. But the described situation does not imply loss of the transmitted IDoc and doesn’t literally mean something is broken in PI/PO – this can mean the IDoc hasn’t actually left a sender system and has never been sent out by it. Below in this blog, I would like to provide few notes regarding meaning of the IDoc status ’03’ and steps how hypothetical depicted case can be analyzed and further root cause analysis performed.
Status ’03’: looking under the hood
As a demo for this blog, I’m going to use outbound IDoc scenario where IDoc is sent out via tRFC – probably, one of mostly common faced communication techniques in SAP outbound IDocs integration. On high level, processing in a sender system involves three layers: application, ALE and communication:
In the most basic scenario, the last step of outbound IDoc processing in ALE layer is dispatching of the IDoc to a port – if this step is successful, status ’03’ is assigned to the IDoc. After the IDoc is sent to a port, by default, ALE layer does not have visibility regarding further processing steps occurring in communication layer, and their outcome.
Now let us assume a delay or error occurred during processing in communication layer – this can happen due to various reasons: sender system overload and lack of CPU or RFC resources causing delay in processing outbound RFC call, failure to establish RFC connection to a receiver system due to misconfiguration, network issues or receiver system unavailability, failure on receiver side during acceptance and handling of inbound RFC call. By default, RFC errors are not propagated back to ALE layer and IDoc remains persisted with status ’03’. Status ’03’ is status indicating success and is depicted using green semaphore colour in monitoring tools, since precisely speaking, ALE layer succeeded with its mission and accomplished all required tasks, finally dispatching the communication IDoc to a selected port:
In order to illustrate this situation, let us send two outbound IDocs – the first one is processed and delivered to a receiver system successfully, whereas before sending the second IDoc, I “broke” the scenario so that IDoc fails when being delivered to a receiver system.
Checking IDoc monitor (here, I used transaction WE02), we can see both IDocs are assigned status ’03’:
But checking tRFC outbound calls (transaction SM58), we can evidence one error that occurred during the second IDoc processing:
To ensure that this is really the call corresponding to the second IDoc, we can explore content of the RFC call (which will contain IDoc content) or alternatively we can fastly verify transaction ID: IDoc status record entry for status ’03’ contains indication of a created RFC transaction ID (tab “Sts details”, field “1st parameter” in transaction WE02), which matches the one observed in tRFC outbound monitor above (field “Transaction ID” in transaction SM58):
As it can be seen, just using IDoc monitoring tools, it is not obvious if the IDoc reached a receiver system or if it got stuck in communication layer of a sender system.
IDoc ALE / RFC status reconciliation
SAP systems are equipped with a standard tool that can be used to improve transparency of status of corresponding RFC transaction correlated with the dispatched outbound IDoc, from ALE layer. This can be accessed using transaction BD75 or program RBDMOIND. There are materials describing this functionality (for example, refer to Setting Dispatch Status to Dispatch OK – IDoc Interface/ALE – SAP Library and SAP Note 1157385). Even though functionality is available in SAP systems, it is not commonly familiar and used by support teams when troubleshooting errors in IDoc scenarios.
Using the mentioned transaction, it is possible to verify if RFC transactions containing outbound IDocs, were processed successfully, or if they are pending processing, in process of execution or in error status in RFC layer. Transaction is not intended to be used for non-RFC port types.
General logic of the transaction is as following:
- Retrieve IDocs with status record ’03’ and non-empty transaction ID, which were last updated on or after date specified on a selection screen of the transaction;
- For selected IDocs, identify corresponding transaction IDs;
- Retrieve tRFC outbound entries for identified transaction IDs;
- Check if RFC entry is found for the IDoc and proceed accordingly:
- If RFC entry is not found for the IDoc (meaning that IDoc was processed successfully in RFC layer and was delivered to a receiver system), add IDoc status entry for status ‘12’ (“Dispatch OK”) to the IDoc,
- If RFC entry is found for the IDoc (meaning that IDoc transmission failed, or is still being processed, or waiting in RFC layer and was not yet delivered to a receiver system), no IDoc status record update occurs and the IDoc remains with status ’03’.
Here is an outlook at IDoc monitor containing same two IDocs, but after execution of transaction BD75 – as seen, the first IDoc (which was successfully delivered to a receiver system) got status ’12’, whereas the second IDoc (which failed due to communication errors) remains in status ’03’:
Details of status record for status ’12’ of the first IDoc provide additional information, where it can be seen that the status was added by already mentioned program RBDMOIND (called by earlier executed transaction BD75):
Optionally, it is possible to enable display of not sent IDoc numbers by using corresponding check box on a selection screen of the transaction. If done so, list of IDoc numbers that were not yet delivered to a receiver system, will be generated in an output.
It shall be noted that the described functionality is only relevant for ports of RFC type. Thus, if a port of another type is used and tRFC layer is not involved, transaction BD75 will not yield to an illustrated outcome and IDocs will remain in status ’03’.
You may be wondering why the described reconciliation mechanism is not enabled out of the box. Reason for this is possible performance impact on a sender system: as mentioned earlier, when being executed, transaction BD75 accesses RFC and IDoc tables. In case of inefficient or time consuming access to these tables (for example, if there are many obsolete RFC entries, or proper cleanup / housekeeping of RFC and IDoc layers is missing, or in case of global database performance issues), performance of this transaction may be degraded. Execution of this transaction at a time of high load on RFC and/or IDoc layer may also cause potential performance issues.
Periodic execution of program RBDMOIND (in case IDoc ALE / RFC status reconciliation is to be conducted on a regular basis) shall also be performed with caution. Intention of making reconciliation near real time and corresponding too frequent execution of program RBDMOIND (e.g. triggered by scheduled background job) may only increase risk of negative performance impact highlighted above for individual manual execution of transaction BD75.
As it was illustrated in the blog, green semaphore colour of the IDoc status in IDoc monitoring tools shall be interpreted correctly – failure to do this may cause misleading steps in further problem root cause analysis and result delays in identification of an actual issue and its resolution.
If an outbound IDoc has status ’03’ and is due to transmission over RFC port, before checking target system, it is recommended to start root cause analysis from looking into RFC layer of a sender system: in case of tRFC, transaction SM58 shall be checked, in case of qRFC (IDoc queue processing) – transaction SMQ1. For systematic mass verification, the described transaction BD75 / program RBDMOIND can be used to facilitate and complement this process.
If an outbound IDoc has status ’03’ and is due to transmission over RFC port, if there is no corresponding RFC entry for it in RFC layer (assuming there was no manual interference and RFC entry was not deleted manually), but execution of transaction BD75 doesn’t result in the IDoc status update to ’12’, there can be purely internal technical error in IDoc update process (for example, IDoc object couldn’t be locked).
If an outbound IDoc has status ’03’ and used port is not of RFC type, transaction BD75 is not helpful in status reconciliation. In such scenarios, it is worth checking monitoring tools and logs of corresponding used communication layer.
Very useful for PI consultants who nearly have no access to SAP ERP system, thanks for sharing. 🙂
Oh man, we learned about the status 03 the hard way. We don't use PI/PO but we have some EDI interfaces (maintained by a third party outside of SAP) and it turned out there were some connectivity problems that caused almost 200 invoices not to reach the recipients.
Our users are trained to monitor for errors but, as you noted, 03 gives you "green light", so everyone thought it was hunky-dory. The hosting provider also was somehow asleep at the wheel and didn't alert anyone of the connection issue. We only found when our AR started calling the customers to collect and found the customers never received the invoices. It's not a situation any company would want to be in!
Thanks for the monitoring tips. We ended up running a custom query to check for status 03 (it conveniently sends an email when such IDocs are found), but those transactions can also be helpful. It's very important to monitor for this status - as you correctly pointed out, it does not mean the whole process was a success.
Thanks for sharing!
Jelena, your experience and the story are really shocking, I can only imagine frustration of both the AR team and customers.
Regarding monitoring and alerting for this kind of issues... Similarly to your case, I commonly evidence that application users and application support team are well trained about usage of IDoc monitoring transactions and check IDocs statuses, but very rarely they go further to communication layer monitoring and thus sometimes they are overestimating IDoc status '03', and also its green colour becomes misleading for them. Besides IDoc ALE / RFC status reconciliation using transaction BD75 or program RBDMOIND (which in some cases, we scheduled as a periodic background job) and subsequent training on interpretation of IDoc statuses '03' and '12', in some projects we also configured CCMS alerting for tRFC and qRFC monitors and corresponding auto-reaction method for them (for e-mails notification). Nowadays this can be embedded into a centralized alerting system (Solution Manager or equivalent), but that time even local CCMS based monitoring already helped to get prompt notification if errors occurred in RFC layer.
Thanks Vadim Klimov. This helped me when I was trying to find out why IDocs haven't reached CPI.
You are welcome, Bhalchandra! Nice this turned to be helpful to you.
Hello Vadim, thanks for the post. Could you please elaborate a little more on this?
“Nowadays this can be embedded into a centralized alerting system (Solution Manager or equivalent)”
Hello Gabriel. Here, I primarily referred to Monitoring and Alerting Infrastructure (MAI) in Solution Manager - more precisely, to Interface and Connection Monitoring (ICMon), and Central Alert Inbox. ICMon can help collecting information on corresponding metrics, and Alert Inbox to get a consolidate view at raised alerts.
Excellent article Vadim. Thank you for sharing this. Exactly what I need to know for my current situation.
You are welcome, Chris. Glad it helped you in your situation.
Useful information Vadim Klimov
Is there a way to find why the IDOCs were dispatched exactly after a day, we wanted to find the exact reason.
The IDOCs were generated on Oct’10th(status 03) but were only dispatched(status 12) exactly after a day. This usually is not the case. Can you please give any inputs that might help us in our case.
It feels there is a confusion in statuses that you refer to: status 30 indicates that the IDoc is ready for dispatch, status 03 - IDoc has been dispatched, and status 12 - dispatch OK (confirmation).
If you observed a delay between statuses 03 and 12, this might be caused by following:
Thanks for the inputs Vadim Klimov
I guess the scenario 2 is our case in here. Just wanted to know if we could find any logs or information related to this to confirm what happened on that day
Tx. BD75 / program RBDMOIND would normally output how many IDocs were found (checked), and how many of those were dispatched successfully or are still "open" (haven't been dispatched, are in progress of dispatching or failed during the dispatching process). It doesn't provide information on specific IDoc numbers that were successful or not, but outputs a number of such IDocs. So you can check job log for RBDMOIND executions to start with.
RFC layer doesn't keep historic data about processed logical units of work (LUWs) - including those containing IDoc messages - so if RFC processing was finally successful (even if it had some errors prior to that), the RFC entry will be gone and you will not be able to get retrospective view on what errors it experienced and why it wasn't processed successfully right at the beginning.
Having said that, I shall note that if you have alerting for RFC errors and receive details of failures in the RFC layer, it might be you will be able to correlate the RFC transaction where IDoc was processed (RFC transaction ID will be indicated in details of the status record for the status 03 in the IDoc) with alerts and errors for it.