Enhancements to Message Processing Log Viewer in the Web Application
A new visualization for Message Processing Logs in the Web Application was released with version 2.38 as a first increment. With version 2.39, we worked on improving this feature to enhance the usability and increase the available information. In the following, I will provide some detail on those improvements.
The affected areas are:
- Record and display trace data for adapters
- Display the Name and Type of the respective Model Step directly in the step list
- Show the traversal path the message processing has taken in the model
- Improved correlation of message processing to the corresponding System Logs
- Download of trace data including headers (improvement with release 2.42)
Record and display trace data also for adapters
In some cases, it might happen that messages are not sent out or received as expected. Those cases were hard to troubleshoot with the tracing feature in case one wanted to check the exact payload as it was transmitted by the sender. The SOAP adapter for instances removes the SOAP specific tags from the transmitted payload when a message is received and only forwards the body for processing. For SOAP based adapters, the received and sent message payloads could already be recorded, however, this was not integrated with the regular tracing; for other adapters, this was not available at all.
With version 2.39, this is now part of the regular tracing feature and it has been extended to more adapters. Message content traced by adapters can be accessed like any other trace data. The payloads are recorded if the log level is set to Trace. Data is recorded before the message is transformed upon reception or after it has been transformed directly before sending. The point at which the trace data was recorded is shown in the section title on the step details page.
The following example depicts an step recorded for a SOAP sender adapter capturing the initially received payload together with its headers:
The following adapters support tracing of sent and received messages:
- SuccessFactors (SOAP and REST only)
- Ariba (with version 2.40)
The goal is to extend this to other adapters as well, however tracing payloads and headers for adapters provides only valuable information for troubleshooting if the payload is actually transformed by the adapter or there are specific headers only provided by the adapter.
The following adapters are not transforming the body nor do they set specific headers, so it is not planned to enable tracing for them:
Note that already deployed Integration Flows containing IDOC, SOAP, and SAPRM adapters have to be redeployed once to enable the trace feature.
Display of Model Step Name and Type in step list
In the first increment of the new log visualization, there was already the option to mark an entry in the list of steps in order to have the respective step in the Integration Flow model highlighted. In order to improve the correlation of steps in the list with the model at a glance, now also the name and type of the Model Step belonging to a certain executed step is displayed directly in the list.
Note that you can specify the name of many integration flow step types in the Integration Flow Designer. If available, this can be done on the General tab of the respective step’s properties.
Show the traversal path the message processing has taken in the model
With the first increment, we also provided a visual indication of the path the message processing has taken in the Integration Flow model. This so-called message traversal path was only available, if the message was processed with log level Trace. The traversal path is marked by decorating the corresponding link with a icon or a icon, if the following step raised an error. This feature has now been extended to log levels Info and Debug. In order to have a visual distinction between steps for which trace data is available and steps for which only the processing information was captured, inverted icons are displayed if there is no trace data available: if the step was processed successfully and if the processing resulted in an error.
When you display a Message Processing Log which was recorded with log level Trace, you might encounter mix cases with respect to the traversal path decoration as some adapters are not supporting the recording of trace data. The link representing the channel would then be displayed with a icon, while other steps would be decorated with the icon.
Correlating message processing with System Logs
System Log Files can be accessed via the respective tile in the Access Logs section. These log files can be used to retrieve more information in case the processing of a message failed with an exception and you assume more context information to be available as part of the system logs. It can be however, that your tenant consists of multiple message processing nodes. In that case, it was cumbersome to find the log which actually might contain additional information for one specific message as there was no indication on which node the message was actually processed.
In order to improve that, we now show the so-called Process ID as part of the Logs section on the message detail view.
This ID can then be used to filter the available system logs as it is part of the system log’s filename.
Download of trace data including headers
In case, you want to download the trace data from the Message Content tab, there is the option to download the payload together with the headers in one zip file. This can be helpful if you want to provide this information to others for troubleshooting, for instance when opening a ticket with SAP. Just keep in mind that the payload can contain confidential data, when forwarding the downloaded information.
If you navigate to the MPL and open the step details of a step carrying trace data, the Message Content tab is available. The Download button is located on the top right hand corner of that tab.
Clicking this button triggers the download of a zip file, which will contain a file with the traced headers as name value pairs and a file containing the traced message body for this step.
I hope, you can benefit from the enhancements presented in this blog post. In case of questions or feedback please feel free to comment on this blog.
Troubleshooting Message Processing in the CPI Web Application
Cloud Integration – Setting the Log Level for Message Processing
Cloud Integration – Enabling Trace for Message Processing
Great to see this in the real world. Look really nice. I guess HTTP is also supported though all headers seems to be removed when using it.
Just added my own video about the functionality.
thanks for the feedback and for making the video!
The HTTP adapter does not support tracing so far. The decoration of the sender channel gives a hint on that as it carries an empty envelope icon instead of the filled one used for steps which were traced. The HTTP adapter does not modify the payload so what you see for the JSON to XML Converter step is exactly the payload as received by the adapter.
However, only dedicated headers are propagated to the IFlow processing. All others are removed in the beginning of the processing to prevent unintended impact unless explicitly allowed (cf. help on Allowed Header(s) configuration property).
I’ll forward your feedback to the corresponding colleagues.
Yes I can see that all other information is removed in the HTTP header. But I guess the heades they can anyway not be used in the processing.
We have been able to use the tracing for enhancing in our testing solution.
For HTTP we will just fetch first information. And it seems like the OData API contains the tracing information.
Is there any plan to include tracing support for when an adapter is used in a Content Enricher step?
Hi Eng Swee,
the Content Enricher step should already be covered. If the adapter is listed above to support trace, the trace information should be collected regardless of the context in which the adapter is used (Content Enricher, Sender / Receiver Channel, Request Reply), if tracing is turned on.
If this is not the case in your scenario, please open a ticket on component LOD-HCI-PI-OP-SRV so that the colleagues could have a look.
Thanks for your response.
In the following scenario, you can see there is a trace envelope icon at the Request-Reply step but not for the Content Enricher.
Anyway, I looked through the trace logs again for when an adapter is used in Receiver channel or Request-Reply, and noticed they do not contain any details of (i) the raw HTTP request sent out, and (ii) the HTTP response from the target server.
To summarise, below are some of the details I'm trying to look for going through the trace/logs:-
i) HTTP response when adapter used in Receiver channel (currently, I have to switch it to a Request-Reply, and add a subsequent step in order to view the response payload)
ii) HTTP response of the Content Enricher (currently, if I view the step after the Enricher, it shows the combined/enrich payload, but I cannot see the actual raw response from the web service call)
Hi Eng Swee,
I was able to reproduce the missing envelope in this case. I have forwarded this issue to the corresponding colleagues to fix this.
Even though the envelope is missing in the model display, the trace payloads were still recorded for me and accessible via the step list in case Log Level TRACE was active.
In the step list in the upper part of the detail screen, there should be entries with traceMessage in the Activity column for the respective step. You can also click on the channel arrow pointing from receiver to the Content Enricher step. Then an information icon appears through which you can retrieve the ID of the step, which you can use to correlate the step entries in the table further; in the example below it was MessageFlow_2.
Could you please check, whether this is also working for you or whether the traceMessage entries are completely missing?
Thanks for your response - appreciate it.
I've tried this again with the same integration flow as before, and do not find the traceMessage activity in the run step entries.
Please refer below where the ID for the channel arrow is MessageFlow_4, but it is not listed in the steps. I can only see the payloads before the Content Enricher call (ServiceTask_3), and before the next step (Groovy script CallActivity_12 in this case).
Not sure if it could be related to adapter type, as I see you are using SOAP adapter for the Content Enricher while I am using SuccessFactors OData V2.
Hi Eng Swee,
it seems that I was too coarse in the statement, that the SFSF adapter does already support tracing. I double checked with the colleagues and the tracing support for the OData variant is still to be implemented. Currently, it is only supported if SOAP or REST is used. That's the reason why you don't even see traced payloads. So in your case, the missing envelope was actually correct.
With respect to the missing envelope in case the payload is available in the Content Enricher step: That should be fixed with version 2.44.
Thanks for the update. Will version 2.44 also include support tracing for SFSF OData variant?
Hi Eng Swee,
unfortunately, there were some issues with the first fix regarding the envelope for the enricher case so that it had to be shifted to 2.45.
Regarding your question: SFSF ODAta tracing is not part of 2.44.