Skip to Content

With the 2.38 version of SAP Cloud Platform Integration an improved representation of the message processing log with embedded integration flow tracing capabilities has been released. This blog gives an overview of the new feature and provides best practices and examples on how to troubleshoot errors occurring during message processing.

Troubleshooting errors occurring during message processing is one of the most crucial tasks for an integration developer. SAP Cloud Platform Integration consolidates all capabilities supporting the user in analyzing such error situations in the “Monitor Message Processing” feature in the Web application of SAP Cloud Platform Integration.

With the 2.38 version of SAP Cloud Platform Integration the “Monitor Message Processing” feature has been extended as follows:

  • Improved log visualization.
  • Log data correlated with integration flow model.
  • Embedded integration flow tracing.
  • Access to integration flow configuration.
  • Better insights into retry scenarios.
  • Log and payload data downloadable.

In the following it is outlined how the various capabilities can be used in the context of a typical troubleshooting use case.

The starting point for investigating on an error occurred during integration flow execution is the “Monitor Message Processing” feature. You can use various search options to retrieve the message you are interested in.

 

When clicking on a message all attached log data such as processing status, log attachments, etc. are displayed in the detailed section of the page. As part of the “Status” section you can find the error message which has been thrown during integration flow execution. In many cases this error message already gives a good indication on what caused the integration flow execution to fail. For example, the error message exposed in the following screenshot indicates that an error occurred during execution of an Apache Groovy script.

 

In case the error message does not indicate on how to resolve the error situation you have the option to continue your investigation by looking into the message processing log. Today’s message processing log is relying on the technical logging of the underlying technologies. In case you are familiar with Apache Camel, Apache CXF, etc. it is a very powerful tool to understand what exactly happened during an integration flow execution.

However, SAP acknowledge that understanding such a technical logging requires expertise in the used technologies such as Apache Camel, Apache CXF, etc. Since not all users are familiar with these technologies it is planned to continuously extend the logging by a more explicit logging per processing step. With that going forward most use cases will not require the user to look into the technical logging provided by the underlying technologies. The technical logging will only serve as backup for experts relying in their work on the technical details.

As part of the 2.38 version of SAP Cloud Platform Integration the visualization of the log data has been evolved. The actual message processing log is displayed in a table format. Each line in the table represents a technical processing step as captured by the underlying technologies. The user can easily locate the step where the error occurred via the  icon. Clicking on the icon opens the error message in a message box.

The log data is correlated with the integration flow model. When selecting a step in the log table the correlated flow-step in the integration flow model is highlighted. This is a powerful tool since it clearly outlines the flow-step in the integration flow model a certain log entry belongs to. For example, it allows to locate the flow-step where the actual error occurred by simply selecting the step flagged with the  icon. 

 

In regards to the troubleshooting use case outlined in this blog it might be required to validate the actual integration flow configuration since a wrong configuration can be the cause for an error during integration flow execution. You can access the integration flow configuration of the selected step in the log table by navigating to the step details using the  icon. To actually change the integration flow configuration you need to switch to the Designer.

Although the message processing log provides detailed logging for an integration flow execution insights into the content of the processed message might be required for certain cases as well. For example, in case the message processing relies on the message content (e.g. mapping, router) it is required to trace the content of the processed message along the integration flow execution.

With the 2.38 version of SAP Cloud Platform Integration such a tracing capability has been introduced in the “Monitor Message Processing” feature. To activate the capability you need to switch to log level “Trace” for the respective integration flow. You can set the log level in the “Manage Integration Content” feature. To generate the trace data in SAP Cloud Platform Integration you need to trigger a reprocessing of the message (e.g. by sending the message again to the SAP Cloud Platform Integration tenant).

Back in the log view of the “Monitor Message Processing” feature you can continue your troubleshooting work. One of the capabilities offered with log level “Trace” is a visualization of the execution path of the message in the integration flow model. This so-called message traversal path is visualized by decorating the integration flow model with the  icon. A red colored icon () indicates the flow-step where an error was thrown during integration flow execution.

 

Today the message traversal path can only be visualized for log level “Trace”. Going forward it is planned to visualize the message traversal data in the integration flow model for the other log levels as well.

To access the content of the processed message you can navigate to the trace data written for a selected step using the icon. In the “Message Content (Before)” tab both the message header and payload are displayed with its state before the message was processed by the respective step.

 

With the 2.38 version of SAP Cloud Platform Integration it is possible to trace the message content along the integration flow execution path. Trace data are written for the processing steps modeled in the integration flow. For adapters used in the integration flow the trace data are not yet exposed in the Web application. In case you are interested in the trace data written by adapters you can download it using the Eclipse Integration Operations feature. Going forward it is planned to support tracing for adapters in the “Monitor Integration Content” feature of the Web application as well.

Finally, in the context of troubleshooting error situations occurring during integration flow execution it is worth to outline two additional rather specific use cases:

(1) In certain situation the message processing runs into an error at a quite early step during inbound processing. At this point in time no message processing log is available. To troubleshoot such error situations you can use the system logs offered for download in the “Monitor System Logs” feature.

(2) In case your integration flow runs into a retry cycle the individual runs of each integration flow execution are displayed in the “Logs” section of the “Monitor Message Processing” feature. Up to 100 runs are listed in the table. You can access the message processing log of each individual run using the  hyperlink offered in the “Log Level” column of the table.

 

I hope you will enjoy using the new capabilities. In case of questions or feedback please feel free to comment on this blog.

 

References:

Cloud Integration – Setting the Log Level for Message Processing

Cloud Integration – System Log Download

Cloud Integration – Enabling Trace for Message Processing

Avoid Storing Payloads in the Message Processing Log, Especially in Productive Integration Flows

How to Avoid Excessive Storage Load Caused by Using MPL Attachments for Message Logging

Cloud Integration – Apache Camel Exchange Properties recorded as Trace Data

To report this post you need to login first.

13 Comments

You must be Logged on to comment or reply to a post.

  1. Eng Swee Yeoh

    Thanks for the update, Maik.

     

    This looks promising, and looking forward to version 2.38 release later this month.

     

    It’s great that we will be getting an easily accessible equivalent of the Camel Message History. This would be a tremendous help especially when an integration flow becomes more complex. We were beginning to develop something similar, but guess we don’t need to now 🙂

     

    Also looking forward to getting more troubleshooting features when the log level is not “Trace”. “Trace” is just temporary, so we don’t always have the luxury to set it in time, as some issues are only highlighted after the fact.

    (0) 
  2. Naresh Dasika

    Nice blog and well explained about the latest update.

    This is one key feature which is missing on Web UI from quite long time and now it is added. Hopefully developers will more prefer to use UI for development of Integration flows after this update.

    Also should see how this works on Web UI when the size of the integration flows are large.

     

    (0) 
  3. Casper Sloth Andersen

    Hi Maik

    This is very cool and a big improvement to the Web UI.

    As one that have used Eclipse and the trace capabilities in there for quite a while there are both things that are better and worse compared to Eclipse. 2 I want to highlight:

    The good: initially viewing the trace seems to be a lot faster (since Eclipse has to download all trace data locally before being able to display).

    The bad: I really miss the possibility to click on the envelopes or steps in the visualisation of the flow (like I do in Eclipse).
    It would be a big improvement if clicking these would highlight the step above so you could jump directly to the header and payload for that step. Right now I seem to only be able to get “technical information” (step ID) and then I have to scroll around to search for that in the list above.
    It seems to only work the other way: clicking on an item in the list, highlights the step in the canvas.

    I hope you are planning to improve this? 🙂

     

    (0) 
  4. Eng Swee Yeoh

    Hi Maik

     

    I’ve been using this tracing feature a bit since the release and I have some feedback and requests.

    1. For simple integrations, it works alright, but once it becomes more complex (lots of steps and local processes), the loading time is very slow, and it doesn’t help that it needs to be refreshed every time you move into a step’s details and back.
    2. For complex integration with lots of run steps, it is hard to navigate between the steps and the graphical display. Currently, what is provided is only 1-direction, click on the run-step, and it is highlighted in the graphical flow. The reverse direction does not work. This makes it hard to find the run step related to an item in the graphical flow. I would like to suggest following enhancements:-
      1. When the graphical element is clicked on, highlight the earliest occurrence (there could be multiple occurrences for steps like Splitter) of the corresponding run step.
      2. Provide a search bar at the top to filter based on Model Step ID. This allows us to search and zoom into a particular step
    3. A graphical element can consist of multiple steps (e.g. 3 steps for a Groovy script) in the Camel Message History. Currently, to view the payload (i.e. Message Content Before), we need to make sure we click on the earliest step for that element. Would be good it any of the step for an element can display the payload.

     

    Regards

    Eng Swee

    (0) 
    1. Holger Himmelmann

      Thanks to Maik for the helpful blog. The new monitoring functionality is indeed a welcome improvement. But I also agree with Eng Swee’s feature requests explained in point 2 and 3.

      Regards, Holger

      (1) 
  5. Maik Keller
    Post author

    Thank you for your feedback! It is indeed planned to allow bi-directional navigation between the log and the integration flow model. Prerequisite for this is to condense the data exposed in the log so that it better maps with the integration flow model. You will see enhancements here in the next increments. Stay tuned…

    (1) 
  6. Former Member

    Thanks Maik for this nice post. I didn’t find any info in the “release notes” for CPI. Could you let us know where we can get the up-to-date info on the new functionalities?

    Regards, Wendy

    (0) 
  7. Maik Keller
    Post author

    Actually all new features shall find their way into the release notes for CPI. In addition we will post additional context information in blogs. You may want to look into the blog from Markus where he outlines some cool new features which will be deployed on all customer systems during the next days.

    (0) 
  8. Eng Swee Yeoh

    Hi Maik , Markus Beier ,

     

    I tried turning on the Trace on an OData Service developed on CPI. However, after executing a message for the OData service, the trace details only contain the run steps list but not the graphical model.

     

    Refer below, where it shows the message “The design model is not available”.

     

    The integration flow was generated by the system when I perform a binding on one of the OData operations.

     

    Regards

    Eng Swee

    (0) 
    1. Markus Beier

      Hi Eng Swee,

      this is not yet supported. Displaying the model for MPLs belonging to “OData Service” artifacts is still in development.

      Best regards,

      Markus

      (0) 

Leave a Reply