Skip to Content
Author's profile photo Bhavesh Kantilal

HCI – Payload Logging using Groovy Scripts – Part 1

Background

Logging during a message processing is always handy be it for run-time debugging during development or during during support when you want to understand what happened during the life cycle of a message.

HCI provides default logging options as described in this blog : HCI First Steps Part 6 – Identifying issues . The tracing options described in this blog are temporary options as the trace is persisted only 10 for Minutes. Try to View Trace post 10 minutes of processing your message and you will get the below error message

/wp-content/uploads/2016/04/1_932789.png

This is obviously not ideal. Are there better options that ensure that the logs are available for consumption post the 10 minutes duration? Those from a PI background would know the advantage of using Trace statements in their user define functions. The trace statements enabled you to understand what happened during the processing of your message. Is there something similar in HCI?

You can insert logging statements in your Groovy Script and Java Scripts. The documentation to this is described here, Defining Scripts

This blog covers a sample using Groovy Scripts but it would work similarly in the context of a Java Script as well.

Groovy Script Example


import com.sap.gateway.ip.core.customdev.util.Message;
import java.util.HashMap;
def Message processData(Message message) {
    def body = message.getBody(java.lang.String) as String;
    def messageLog = messageLogFactory.getMessageLog(message);
    if(messageLog != null){
        messageLog.setStringProperty("Logging#1", "Printing Payload As Attachment")
        messageLog.addAttachmentAsString("ResponsePayload:", body, "text/plain");
     }
    return message;
}










The above script has 3 critical components,

Code Usage
def messageLog = messageLogFactory.getMessageLog(message);
  • Get access to the Message Processing Log
messageLog.setStringProperty(“Logging#1”, “Printing Payload As Attachment”)
  • Write a String Property to the Message Processing Log
  • In the message processing Log, the property Logging#1 is written with the content “Printing Payload as Attachment”
  • In this case we have hardcoded the “Printing Payload as Attachment” String. You can also set this to the Message Header, Property and Payload as you require.
  • [Update: Jul 27, 2016 – Added statement as per recommendation from Stefan Boller  in the comments section] This method setStringProperty should be used only for short Strings (one or a few words). For longer, structured documents like payloads method addAttachmentAsString should be used. See next row for this method.
messageLog.addAttachmentAsString(“ResponsePayload:”, body, “text/plain”);
  • Write an attachment to the Message Processing Log
  • The input message payload is inserted as an attachment into the Message Processing Log.

Example Script Usage in a Integration Flow

/wp-content/uploads/2016/04/2_932840.png

This is a simple SOAP to SOAP Scenario with the Script to log the response of the SOAP call to the Message Processing Log as an attachment. The Script listed above is used in a Integration Flow after the Request/Reply step.The expected result is that the Request-Reply step response is written to the message processing log.

Testing of the Integration Flow with the Groovy Script

Eclipse

Trigger your Integration flow, and then proceed to the message processing log in Eclipse. Below will be seen.

/wp-content/uploads/2016/04/3_932841.png

The Logging#1 contains the Text that was printed as a String in the Groovy Script.

As mentioned previously, while we have hard coded this in our example to a constant string, this can also be any of the Message Headers, Message Payload and Message Properties as required.

The Attachments contains a Hyper Link with the name of the Attachment as mentioned in our source code. Clicking on this hyperlink, will lead to the actual payload.

/wp-content/uploads/2016/04/4_932842.png

WebUI

Likewise, when monitoring your flow over the WebUI, the same info is visible. In fact the WebUI provides the Message Processing Log and Attachments separately making it easier to navigate and view the specifics as required,

/wp-content/uploads/2016/04/5_932852.png

/wp-content/uploads/2016/04/6_932853.png

/wp-content/uploads/2016/04/7_932854.png

Sequence of Printing of Logs

Eng Swee Yeoh had pointed to a very important concept in terms of how the Logs are printed. The documentation link pointed previously has the below info,

Note that the properties provided by the script step are displayed in alphabetical order in the resulting message processing log (MPL). That means that the sequence of properties in the MPL does not necessarily reflect the sequence applied in the script.

What this basically means is if your Groovy Script has multiple print setStringProperty, these are not printed in the logs in the same order as they get executed in the Message Processing Log. Infact, they are printed alphabetically.

For example, if your Grrovy script has the statements,


messageLog.setStringProperty("DEF", "Printing Payload As Attachment1")
messageLog.setStringProperty("ABC", "Printing Payload As Attachment2")






The message processing Log will print the output in Alphabetical order instead of the sequence in which it is executed, i.e,  for the example listed above, ABC is printed 1st, followed by DEF as per below image.

/wp-content/uploads/2016/04/9_933822.png

When implementing your Groovy Script to use print in the Message Processing Log, it is recommended that you ensure that the String Properties have the Key’s as Log#1, Log#2 and so on to that the Printing happens in the same order as execution.

Final Note

Like everywhere else, there are performance tradeoff’s between Logging multiple payloads and not logging the same and hence this should be used with required design considerations. It is recommended that the Groovy Logging described in this blog is used in combination with Externalized Parameters to enable / disable logging of Payloads for an Integration Flow. Your externalized parameters can be set to a Logging true/false which is then checked within your Groovy Script and then the Data Logged as required.

The steps to externalize your logging is described in the next blog of this series : HCI – Payload Logging using Groovy Scripts – Part 2 – Use Externalized Parameters

Assigned Tags

      21 Comments
      You must be Logged on to comment or reply to a post.
      Author's profile photo Eng Swee Yeoh
      Eng Swee Yeoh

      Hi Bhavesh

      I remember you asking about this, then figuring it out yourself. Nice to see it in a blog post now 😉

      One thing I'd like to add, I remember reading that the logs created by this approach appear in alphabetical order and not in processing sequence. Therefore having a counter of some sort would help in generating logs that make sense in a chronological order. Of course the logic for such a counter (static or dynamic) is up to whoever wants to use this, but just thought I mention it.

      Regards

      Eng Swee

      Author's profile photo Bhavesh Kantilal
      Bhavesh Kantilal
      Blog Post Author

      Hello Eng Swee,

      I did try this and in both Eclipse and in WebUI the message processing log contains the Logs in sequence of which step is executed. There is no dependency on the alphabetical sequence.

      Basically if we introduce the Groovy Script twice in the above Integration Flow with just changes to the Key's, the message processing log will show these as separate sections and in order of execution of the script in the Integration Flow.

      /wp-content/uploads/2016/04/8_933062.png

      Is this what you are talking about or am i missing something?

      Regards

      Bhavesh

      Author's profile photo Eng Swee Yeoh
      Eng Swee Yeoh

      Hi Bhavesh

      I did some digging and found the following in the online documentation link you referenced above.

      /wp-content/uploads/2016/04/order_933792.png

      To be honest, I didn't get around to testing this when I was evaluating the trial tenant, so I'm not sure of the behavior (and I no longer have access to a tenant now 🙁 ).

      If you use setStringProperty() more than once (either consecutively or with other logic in between) within the same Groovy script, would it appear in separate sections as well?

      By the way, the externalized parameter approach to control the logging is a really good idea. Maybe how it can be used within a script might be worthy of a blog? 😉

      Regards

      Eng Swee

      Author's profile photo Bhavesh Kantilal
      Bhavesh Kantilal
      Blog Post Author

      Hello Eng Swee,

      You and the documentation are right. If within the same script, there are multiple setProperty steps, they are printed in Alphabetical Order than than in the sequence they are executed. Will update the blog accordingly so users are aware of the same. Great Catch!

      Will also publish a extension of this blog for the Externalized Parameters usage.

      Regards

      Bhavesh

      Author's profile photo Bhavesh Kantilal
      Bhavesh Kantilal
      Blog Post Author

      Hello Eng Swee,

      Here is the next blog to externalize this as requested! Happy Reading and thanks for prodding 🙂

      HCI - Payload Logging using Groovy Scripts - Use Externalized Parameters

      Regards

      Bhavesh

      Author's profile photo Eng Swee Yeoh
      Eng Swee Yeoh

      Fantastic, Bhavesh!

      Thanks for complying to my whims and fancies at such break-neck speed! 😉

      As HCI development continues to pick up, these will definitely be invaluable reference materials to the community.

      Thanks again,

      Eng Swee

      Author's profile photo Bhargava Krishna Talasila
      Bhargava Krishna Talasila

      Hi Bhavesh,

      Great blog...!

      Thanks for sharing 🙂

      Author's profile photo Former Member
      Former Member

      Hello Bhavesh and Eng Swee,


      Could you please help me to fix a successfactors asynchronous query issue? Normal query works fine but when I try to add ${property.xyz} in the where clause, it doesn't get the value of that property and just send the property name as a property value so the query doesn't work. Please let me know if you know anything about this issue I will really appreciate.


      Note: It works fine with the synchronous query but Adhoc entity doesn't support it.

      Thanks.

      Author's profile photo Michael Appleby
      Michael Appleby

      Unless you are asking for clarification/correction of some part of the Document, please create a new Discussion marked as a Question.  The Comments section of a Blog (or Document) is not the right vehicle for asking questions as the results are not easily searchable.  Once your issue is solved, a Discussion with the solution (and marked with Correct Answer) makes the results visible to others experiencing a similar problem.  If a blog or document is related, put in a link.  Read the Getting Started documents (link at the top right) including the Rules of Engagement. 

      NOTE: Getting the link is easy enough for both the author and Blog.  Simply MouseOver the item, Right Click, and select Copy Shortcut.  Paste it into your Discussion.  You can also click on the url after pasting.  Click on the A to expand the options and select T (on the right) to Auto-Title the url.

      Thanks, Mike (Moderator)

      SAP Technology RIG

      Author's profile photo praveen bandaru
      praveen bandaru

      Hi Bhavesh,

      Thank you for sharing the info, it is very useful. Is it possible to save the payload as local desktop file. The attachment is saving on HCI server, but I am trying save this as a local file to send it to team members. Is there any way in HCI I can achieve this?

      Thank you,

      Praveen

      Author's profile photo Bhavesh Kantilal
      Bhavesh Kantilal
      Blog Post Author

      Hello Praveen,

      The attachment can be downloaded locally and saved locally. Below is what you need to do,

      1. Click on the Attachment HyperLink in your Message Processing Log/wp-content/uploads/2016/05/59_945310.png
      2. Attachment opens in a View called "Attachment"
      3. Press Save Button in this to download the attachment / payload as a local file./wp-content/uploads/2016/05/60_945338.png

      Regards,

      Bhavesh

      Author's profile photo Stefan Boller
      Stefan Boller

      Hi Bhavesh,

      nice blog post. Can you please add the following to it to avoid issues in the future:

      The method setStringProperty shall be used only for short Strings (one or a few words). For longer, structured documents like payloads method addAttachmentAsString shall be used.

      This is because the String properties are inlined into the Message Log, whereas the attachements are stored in a separate table.

      Dr. Stefan Boller
      Development Architect, HANA Cloud Integration,
      SAP SE
      , Dietmar-Hopp-Allee 16, 69190, Walldorf, Germany

      Author's profile photo Bhavesh Kantilal
      Bhavesh Kantilal
      Blog Post Author

      Hello Stefan,

      This has been done!

      Regards

      Bhavesh

      Author's profile photo Amber Badam
      Amber Badam

      Excellent Blog. Very handy in logging the messages. Keep up the nice work.

      Regards,

      Amber

      Author's profile photo Francois Maas
      Francois Maas

      Hi Bhavesh

      This is a very good explanation, having used it myself quite a bit.  One thing I am uncertain about is the retention period for storing these messages in HCI.  Are you aware of any configuration or settings for local integration processes or globally on the instance that we need to set to manage the retention period for logged messages in the MPL?

      Thank you.
      Francois

      Author's profile photo Wolfram Siefke
      Wolfram Siefke

      Logged messages (Attachments) of an MPL have the same retention time as the MPL itself. By default this is 90 days which will be reduced to 30 days in the near future. The deletion of outdated entries happens with Cleanup Monitoring Data job which is normally auto scheduled with the mentioned default retention period.

      Regards;

      Wolfram.

      Author's profile photo Pavithra Thiagarajan
      Pavithra Thiagarajan

      Hello Wolfram,

      Is there a way to increase the default retention period? thanks

       

      Regards

      Pavithra

      Author's profile photo Wolfram Siefke
      Wolfram Siefke

      Risk of OOM failure on worker nodes:

      The described approach is nice to put dedicated information from the message object into MPL or apart of it. However using this excessively can be the root cause for OOM failures on the worker nodes. Especially

      def body = message.getBody(java.lang.String) as String;

      - depending on the data type of the body - can result a String which is much larger in size than the payload itself. But also Properties and Headers can reference objects which yield huge String representations in the end. So please consider following recommendations when making use of this script based logging approach:

      Only log dedicated properties and headers from which you know they can be converted to String with no danger. The same holds for the payload instance itself.

      Make sure that this logging is deactivated on the productive instances to avoid unintended risks of failure. Unnecessary logging information may also have an impact on the processing performance as well.

      Regards,

      Wolfram.

      Author's profile photo Bhavesh Kantilal
      Bhavesh Kantilal
      Blog Post Author

      Hello Wolfram,

      Apologies for not responding back earlier. Would you like me to incorporate these comments into the content of the blog. Considering this is from the Product Management team and this blog could be getting multiple hits, I would be happy to add your recommendation. Thanks!

       

      Regards,

      Bhavesh

      Author's profile photo Former Member
      Former Member

      Thank you for this valuable documentation.

      I may have something to add to bypass the alphabetical sorting of attachments. The idea is to add a timestamp as a prefix to the attachment name. Please click here for more details.

      Regards,

      Amine.

      Author's profile photo Robin Vezina
      Robin Vezina

      In which scenario would the returned messageLog object would be null in “def messageLog = messageLogFactory.getMessageLog(message);” ? The null checking appears noisy and unnecessary.

      Is the javadoc available for messageLogFactory and messageLog ? I tend to not trust 1 year old blog as a reliable source of information.
      Also, having download access to the library itself would enable static code scan and improve quality in general.