Message Staging and Logging Options in Advanced Adapter Engine of PI/PO 7.3x/7.4
Intro
One of features that were available in ABAP stack of PI 7.1x systems and that its Java stack was lacking is customizable behavior of message persistence for various processing steps. Transition towards Java-only processing in PI brought special attention to this topic due to major requirements focusing on more precise monitoring of messages processed in Advanced Adapter Engine. One of very common use cases is the requirement to check payload of the processed message before and after mapping execution (for example, in order to identify correctness of the mapping execution for the particular message executed at runtime). As a result, several significant and very useful enhancements in this area – namely, customizable message staging and logging in Advanced Adapter Engine – were provided in PI 7.3x and extended in PI 7.31.
In this blog, I would like to summarize various message staging and logging options that are available in Advanced Adapter Engine of PI 7.3x.
Overview
Following message staging and logging options are available in Advanced Adapter Engine of PI 7.3x:
- Global staging/logging;
- Scenario-specific staging/logging using advanced settings of Integrated Configuration;
- Scenario-specific logging using adapter module in communication channel.
In PI 7.30, message staging customization is only possible globally for the entire Advanced Adapter Engine – meaning that all scenarios running in the corresponding Advanced Adapter Engine are affected by respective staging configuration. Customizable global message logging and scenario-specific staging/logging options are not available in PI 7.30 yet.
In PI 7.31, customizable global logging (originally for synchronous messages) mechanism as well as scenario-specific staging and logging options were introduced making it possible to customize staging and logging of messages in Advanced Adapter Engine differently for specific scenarios using advanced settings of Integrated Configuration or a dedicated adapter module configured in the respective communication channel.
In SP04 for PI 7.31, functionality of message logging is extended making it possible to use logging of asynchronous messages.
Before proceeding with discussion of various staging and logging options, let’s firstly highlight major differences between these two mechanisms of the message persistence in Advanced Adapter Engine.
Purpose of message staging is persistence of an asynchronous message at a particular runtime processing step (stage) for later editing/restart of the message from the persisted version as well as for displaying persisted (staged) message.
Purpose of message logging is persistence of any kind of an asynchronous/synchronous message at a particular runtime processing step for later displaying of the persisted (logged) message. Please note that editing or restart of the logged message is not possible.
Message logging is a more lightweight mechanism in comparison to message staging since there is less information persisted during logging (only information required for message displaying) than during staging (all information necessary for message displaying and restarting) as well as because during staging the persisted message is put to the queue and retrieved from it for processing steps where staging applies. Thus, in case of pure monitoring purposes, staging will bring more overhead to message processing, so logging is recommended in this case.
Redundant staging and logging configured for several (many) processing steps should be avoided since it brings negative impact to message processing time in Advanced Adapter Engine as well as additional load on database resulting in increased execution time of the scenario in PI and increased database growth rate.
Global staging/logging
Message staging and logging behavior is configured adjusting corresponding properties of the J2EE service XPI Adapter: XI:
- Property xiadapter.stage.conf – for staging configuration;
- Property xiadapter.logger.conf – for logging configuration.
For both mentioned properties, value syntax is composed by specifying processing step(s) and staging/logging mode (condition) for each of specified processing step. If messages should be staged/logged at several processing steps, respective step/mode pairs are delimited with comma ( , ).
Possible processing steps and modes that are relevant for staging are provided in tables below:
| Processing Step |
Description |
|---|---|
| BI | After sender adapter processing, but before inbound schema validation |
| VI | Before scenario look-up step (check if scenario is local Advanced Adapter Engine based or dual-stack based involving Integration Engine) |
| MS | Before mapping step |
| AM | After mapping step, but before outbound schema validation |
| VO | After outbound schema validation |
| Mode |
Description |
|---|---|
| 0 | MODE_NO_STORE: No message persistence (staging) |
| 1 | MODE_STORE_ON_ERROR: Message is persisted only if its next processing step (stage) throws an exception |
| 2 |
MODE_STORE_AND_CONTINUE: Message is persisted and processing is continued with the next step. If its next processing step throws an exception, the processor reschedules the message from this storage and returns Note: the mode is deprecated in PI 7.31 due to performance inefficiency during message persistence |
| 3 | MODE_STORE_AND_RETURN: Message is persisted and rescheduled, processing is continued with the next step |
For logging, only processing steps BI, MS and AM are relevant.
Mode values that are relevant for logging are derived from ‘basic’ 4-bit binary values by summarization of respective ‘basic’ binary values and its further conversion to a decimal value. Relevant ‘basic’ values are enumerated in the table below:
| Binary Value |
Description |
|---|---|
| 0000 | No logging |
| 0001 | Asynchronous messages are logged |
| 0010 | Synchronous messages are logged |
| 0100 | Onlу еrrοr mеѕѕаgеѕ аrе lοggеd |
| 1000 | Logging with cοntеnt hіdіng (payload is hidden) |
Sample custom configuration of global message staging is provided below:
In this example, asynchronous messages are persisted (staged) in two versions – before mapping and after mapping.
In Message Monitor, the persisted versions of the message can be observed (highligted on the screenshot below):
Please also note that during staging, the message was put into the queue and retrieved it multiple times (corresponding to amount of processing steps for which staging was configured and should be conducted for the processed message). This can be noticed when examining audit log of the message:
Similarly, sample custom configuration of global message logging is provided below:
In this example, asynchronous messages are persisted (logged) in two versions, at the same processing steps as in the global staging example – before mapping and after mapping.
In Message Monitor, the persisted versions of the message can be observed (highligted on the screenshot below):
In logging, there is no multiple operations of queue write (put) / read (retrieve) due to difference in nature of the logging mechanism in comparison to the staging mechanism:
Absence of additional queue write/read operations results in less overhead caused by logging than by staging – the effect that was already mentioned earlier in this blog.
Scenario-specific staging/logging using advanced settings of Integrated Configuration
Sometimes it is necessary to apply specific (more or less precise) staging or logging for the particular scenario – meaning that staging/logging configuration for the corresponding scenario should differ from the globally defined one (described in the previous section of the blog). In order to fulfill such a requirement, configuration of scenario-specific staging and logging can be performed using the tab ‘Advanced Settings’ of the Integrated Configuration object in PI 7.31. On this tab, the one can choose at which processing steps staging or logging should be activated for which modes.
Valid processing steps that can be chosen for staging and logging using advanced settings of Integrated Configuration are the same as those applicable for global staging and logging configuration and described earlier.
For staging, following modes are valid:
- None (no staging);
- Store (stage always);
- Store on error (stage only in case of error at subsequent processing step).
For logging, following modes are valid:
- None (no logging);
- Log (log always);
- Log on error (log only in case of error at subsequent processing step);
- Log without payload (log always, payload is hidden);
- Log on error without payload (log only in case of error at subsequent processing step, payload is hidden).
Sample configuration of combined options of staging and logging of Integrated Configuration is provided below:
In this example, messages (including their payloads) are persisted (staged) right after sender adapter processing and are further persisted (logged) before mapping and after mapping.
In Message Monitor, the persisted (both staged and logged) versions of the message can be observed (highligted on the screenshot below):
Scenario-specific logging using adapter module in communication channel
In PI 7.31, a SAP standard adapter module MessageLoggerBean is provided – this module can be inserted in the adapter module sequence of the corresponding communication channel.
Fully qualified adapter module name is localejbs/AF_Modules/MessageLoggerBean .
Adapter module parameters and their valid values are enumerated in the table below:
| Parameter Name |
Description | Parameter Values |
|---|---|---|
| log.condition | Condition (mode) at which logging should be activated | Integer value derived from following ‘basic’ values:
|
| log.location | Optional parameter that specifies processing step within the adapter module processing sequence where the message should be logged | Name of the processing step within the adapter module processing sequence. If not specified, location default is used. |
Sample configuration of the adapter module processing sequence of the communication channel that utilizes the described adapter module is provided below:
In this example, asynchronous messages (including their payloads) that are processed by the receiver communication channel’s SOAP adapter bean are persisted (logged).
Below is an extract of the audit log of the processed message were log entries reflecting execution of MessageLoggerBean are highlighted – as seen, logging mode (controlled by log condition adapter module parameter) is indicated:
In Message Monitor, the persisted version of the message can be observed, version of the message is controlled by log location adapter module parameter (highligted on the screenshot below):
Outro
Described staging and logging options can be employed in combination making staging and logging in Advanced Adapter Engine even more flexible.
For example, the below persisted versions of the processed message are outcome of joint usage of:
- Global staging before mapping;
- Scenario-specific logging after mapping for an utilized Integrated Configuration object;
- Adapter module MessageLoggerBean in a receiver communication channel for SOAP adapter bean log location.
References
There are also other very nice materials on SCN that cover message staging and logging functionality available in Advanced Adapter Engine of PI 7.3x – namely, it is highly recommended to go through following blogs to get additional information on the topic:
- PI/XI: PI 7.3 message version persistence for ICO integration flows – teaser, the blog written by Michal Krawczyk: http://scn.sap.com/community/pi-and-soa-middleware/blog/2010/10/18/pixi-pi-73-message-version-persistence-for-ico-integration-flows–teaser
- Scenario specific message staging and logging in PI 7.31 AAE, the blog written by Nageshwar Reddy: http://scn.sap.com/community/pi-and-soa-middleware/blog/2012/07/26/scenario-specific-message-staging-and-logging-in-pi-731
- Message Versioning in the Advance Adapter Engine of PI 7.30, the blog written by William Li: http://scn.sap.com/community/pi-and-soa-middleware/blog/2012/01/03/message-versioning-in-the-advance-adapter-engine-of-pi-730
Hi Vadim,
Excellent Blog. Difference between staging and logging clearly outlined.
We are on PI 7.30 I can only see xiadapter.stage.conf in NWA and not the log property.
Is there any SP/Release restriction.
Many Thanks,
Sudharshan N A
Hi Sudharshan,
Customizable logging isn't possible in 7.3. In 7.3, only customizable global staging is available, global customizable logging was introduced in PI 7.31. Thank you for this comment - I updated the blog with this remark so that now it shouldn't be confusing. Strictly speaking, customizable message persistence was introduced in PI 7.3, but many significant improvements and nice features in this area were actually implemented in PI 7.31.
Regards,
Vadim
Hi Vadim,
Good blog!
We have an problem with a async/sync bridge using an adapter module (AF_Modules/RequestResponseBean) in 7.31. We don't find the error message in the log viewer in the nwa, only in the message monitoring of pi we got this error:
exception caught with cause
com.sap.engine.interfaces.messaging.api.exception.MessagingException:
while trying to invoke the method
com.sap.aii.af.sdk.xi.mo.xmb.DynamicConfiguration.get(java.lang.String,
java.lang.String) of an object loaded from local variable 'dynConfig'
We are setup your log module in our adapter, but we don’t see the error message.
Have an idea for us?
By the way check your picture, same are not display correct..
Regards,
Stefan
Hi Stefan,
Thank you for pointing to the issue with pictures - I re-published the blog and now they should be back again.
As for the error you get... MessageLoggerBean is intended to log the version of the message for the particular step (module) executed in the adapter module processing sequence - thus, if some error happened after logged version of the message, this will not be caught by the logger module. Instead of it, you can use XPI Inspector tool and use communication channel based inspection to collect details of the faced error - namely, full callstack that caused this error and also error details and context. For the error itself, I would propose you to create a thread on SCN for it so that we (and other SCN community members) could try to sort it out and help you in solving the problem.
Regards,
Vadim
Hi Vadim Klimov,
Very detailed blog. Thanks.
I also had a blog written on staging and logging (PI 731): Here is the link
http://scn.sap.com/community/pi-and-soa-middleware/blog/2012/07/26/scenario-specific-message-staging-and-logging-in-pi-731
Hi Reddy,
This is surely a fair comment, thank you. I updated the blog with references and included links to SCN materials on message logging/staging in AAE 7.3x topic.
Regards,
Vadim
Hi Vadim,
Great blog. Very detailed. I just wanted to share something I noticed...
In the case of multiple Receivers or multiple Interfaces after interface determination, the MS stage will always get staged for each new message id, regardless of global or specific settings.
In global I had MS=1 and for specific I had MS=None.
In the case of single receiver and single target interface it works as expected.
But when conditions match for multiple receivers or multiple interfaces, the message gets staged at MS for each split message.
Jonathan
Nice Blog..Cheers!!
Well explained.
Very well Explained.Fantastic blog 🙂
Cheers
Pavan
Thanks, excellent blog!
Is there any chance you know me where to find the "Advanced Settings" Tab of the ICO in an IFlow?
I could edit the generated ICO, but editing generated objects is a very bad idea.
Hello Elias,
You don't need to change generated iFlow properties in Integration Directory. In order to access iFlow-specific logging and staging parameterization, in NWDS, please open the corresponding iFlow, go to view 'Properties' and then to tab 'Runtime Configuration':
If you don't see the the view 'Properties' in the SAP Process Integration Designer perspective, please go to Window > Show View > Other... and select General > Properties there.
If Properties view looks different than the one provided on the screenshot above, please ensure that you didn't select any of sub-objects used in the iFlow (like business component, communication channel, mapping, etc.), but the iFlow object is selected (e.g. click on empty area in canvas of the iFlow representation in Model Configurator).
Regards,
Vadim
Hi Vadim,
you were right, the "Properties" view was not active. Displaying it solved the problem.
Thank you!
Regards
Elias
Excellent Blog...
Great blog. Just a question. How we can define the persistence time of a message as usually defined in SXMB_ADM?
Auto responding 🙂 from How to persist the messages on Adapter engine?
Asynchronous messages. Retention period is adjusted using propertiesxiadapter.inbound.persistDuration.default (for inbound messages) andxiadapter.outbound.persistDuration.default (for outbound messages) of J2EE service XPI Adapter: XI (com.sap.aii.adapter.xi.svc). Values of both of them are in milliseconds, the default value is86400000 which is exactly 24 hours. Please note that in addition to setting these properties, respective archiving and deletion jobs periodicity should be checked in Adapter Engine.
Synchronous messages. Synchronous messages are not persisted in database and are only stored in-memory of the Message Store - here total amount of stored messages matters (only last X messages are stored). Amount of synchronous messages that are kept in Message Store is regulated using property messaging.syncMessageRemover.messageCount of J2EE service XPI Service: Messaging System (com.sap.aii.af.ms.svc). Each given period of time (regulated by propertymessaging.syncMessageRemover.checkInterval (in minutes) of the same J2EE service), amount of syncronous messages kept in Message Store is counted and corresponding messages are cleaned out of it by the respective thread (so, speaking differently, this property specifies a sleep time for this thread). Please note here that in the time period between neighbouring checks, there can be a case when amount of synchronous messages is more thanmessaging.syncMessageRemover.messageCount: e.g. in case of high load or large value ofmessaging.syncMessageRemover.checkInterval.
Just one additional remark to your message, Xavi.
With 7.31 SP12 (7.40 SP07) configuration of individual/scenario-specific archiving & deletion retention periods feature is available. 1960448 - Configure message retention period per interface
Excellent Blog 🙂
Hi Vadim,
thanks for the very detailed and interesting blog.
I just tested the Module MessageLoggerBean in one of our communication channels (SOAP Adapter, Transport Protocol HTTP Axis).
It works fine when I put it at the beginning of the module order and I see the entry in the message editor versions.
But when I put the MessagerLoggerBean after the AF_Adapters/axis/HandlerBean (after module key wssec or trp, were the logon Information for the called web service is set) the additional logging entry is not being displayed in the message editor anymore. The message protocol says that the LoggerBean is processed.
What am I doing wrong?
Best regards,
Stefan
Hi Stefan,
Generally speaking, MessageLoggerBean does not depend on specific adapter modules before/after which it is called. I would suggest to perform a trace of the affected Axis communication channel (for example, using XPI Inspector) and get more details regarding execution of MessageLoggerBean. In the trace, please pay special attention to entries corresponding to invocation of the method MessageLoggerBean.performLogging() since actual logging logic is called (or not called) from there and the method writes important trace entries indicating if message logging takes place and if not, then what the reason for this is.
Regards,
Vadim
I am new to PI/PO, I would like to know if we can use log4j type logging to add customized logging in PO.
to finish off my comment, what Iam looking at is add customized logging at sender channel, receiver channel using modules and use those log info to external system like splunk where I can view for different purposes.
Hi Ashok,
The entire message staging and logging functionality is based on usage of specifically dsigned tables as a persistence layer - hence, usage of alternative logging frameworks is not of much use here.
Generally speaking, application server Java comes with SAP proprietary implementation of the logger framework (SAP Logging API). There are certain possibilities to customize its behaviour and utilize it in conjunction with alternative log destinations - as a result, log records can be shipped not only to default log and trace files, but to some other arbitrary targets (remote servers, messaging systems, databases, log processing systems, etc.). It shall be noted that fulfilment of such requirement will require custom development. As an example for one of such use cases, you can have a look into the following blog.
In case you need to utilize alternative logging frameworks and their implementations, it is also possible, but will again require custom development - you can find examples of usage of alternative logging frameworks in custom Java applications deployed to SAP application server Java, but that is not related to staging/logging functionality described here.
Regards,
Vadim
I am new to PI/PO,I want to keep synchronous message more longer.I think I can configue this parameter,but it didn't work.What's "active Message Logging"?
Thanks.
I know this blog is nearing it's tenth birthday 🙂 However, since this is one of the first results when you google logging / tracing and PI I'll give it a shot here:
We have a synchronous scenario which ocassionally fails (nothing I can do about it, it's the receiver side which is out of my sphere of influence.
Is there a way to ommit / suppress logging for failed synch messages on a scenario level completely? I know it would be wiser to get things fixed, but I like to not have false positive alerts and blow up the database with unneccesary logging entries in BC_MSG_LOG table.
Cheers
Jens