External Collection of Message Processing Statistics from Advanced Adapter Engine of PI/PO
Intro
SAP PI system is already shipped with extensive toolset of monitoring applications that can provide aggregated statistics on messages that have been processed in Advanced Adapter Engine (from message overview and performance perspectives). In addition to built-in tools, the one can employ functionality of Solution Manager for centralized monitoring of the PI system in general and statistics of message processing in particular. But there are some cases, when these tools are not sufficient and it is necessary to retrieve statistics of message processing in Advanced Adapter Engine of the SAP PI system for external usage – e.g. in case the consumer of this data is the external monitoring application.
To achieve this requirement, it is possible to query two standard servlets of Advanced Adapter Engine that can provide necessary information via HTTP (thus, accessing PI host via J2EE HTTP port – by default, 5<instance number>00). In this blog, I will provide an outlook at their functionality and use cases. The discussed servlets are:
- MessageOverviewQueryServlet
- PerformanceDataQueryServlet
Both servlets are exposed by Message Display Tool application of Advanced Adapter Engine:
Requests for these servlets are constructed using input parameters (passed as GET parameters in the URL) and responses are rendered in a form of XML document that can be traversed and analyzed conveniently by the caller application. XSD schemas of responses of the mentioned servlets are accessible via URI /mdt/monitor/<servlet name>.xsd.
MessageOverviewQueryServlet
The servlet is used to access message processing overview data externally. In PIMON, this is data displayed in Message Monitor -> Message Status Overview.
The servlet is accessible via URI:
/mdt/messageoverviewqueryservlet
XSD schema of servlet response is accessible via URI:
/mdt/monitor/MessageOverviewQuery.xsd
Semantics of the constructed URI for querying servlet is:
/mdt/messageoverviewqueryservlet?component=<XIComponentName>&view=<ViewNameKey>&begin=<IntervalBeginTime>&end=<IntervalEndTime>[&detailedStatus=<true|false>]
When querying the servlet, following parameters can be used:
| Parameter | Description | Value format / Possible values |
|---|---|---|
| component | PI component. Components can be checked in Component Monitor. Examples are central Adapter Engine and non-central Adapter Engine, central Integration Engine and local Integration Engines (proxy runtimes of connected application systems) | |
| view | View key. The view defines a set of retrieved attributes and sorting principles for queried message overview statistics |
R_ENTRY_VIEW_XPI S_ENTRY_VIEW_XPI SR_ENTRY_OVERVIEW_XPI SR_ENTRY_VIEW_XPI |
| begin | Time period begin time stamp | YYYY-MM-DD hh:mm:ss.0 |
| end | Time period end time stamp | YYYY-MM-DD hh:mm:ss.0 |
| detailedStatus | Indicator for showing status groups / detailed status values |
true false |
If the servlet is called without parameters, the response will contain a list of components that can be queried.
If the servlet is called and a valid value for the component is passed, the response will contain a list of view keys and time periods per period type for which message overview data is available for the specified component. Following period types are supported:
- HOURLY: message overview data for last 12 hours, aggregated per 1 hour;
- DAILY: message overview data for last 7 days, aggregated per 1 day;
- WEEKLY: message overview data for last month, aggregated per 1 week;
- MONTHLY: message overview data for last 12 months, aggregated per 1 month;
- ANNUALLY: message overview data for several years, aggregated per 1 year.
If the servlet is called and valid values for component, view and time period are passed, the response will contain message overview data for processed messages – namely:
- Scenario attributes (scenario identifier, sender and receiver communication components, interface);
- Message counter per message status (for example, for statuses ‘System error’, ‘Waiting’, ‘To be delivered’, ‘Delivering’, ‘Holding’, ‘Successful’, ‘Canceled with errors’ for Advanced Adapter Engine).
Example request (URI part):
/mdt/messageoverviewqueryservlet?component=af.po1.nced60229921a&view=SR_ENTRY_VIEW_XPI&begin=2014-02-24 01:00:00.0&end=2014-02-25 01:00:00.0&detailedStatus=true
Response (extract depicting two message overview data set entries):
Names of respective entry elements of data rows are provided at the beginning of the servlet response XML document:
The servlet usage is described in the SAP Note 1400785. The servlet is available as of PI release 7.0. Please consult the SAP Note for minimum SP/PL level requirements for the particular PI release.
PerformanceDataQueryServlet
The servlet is used to access message processing performance data externally. In PIMON, this is data displayed in Performance Monitor.
The servlet is accessible via URI:
/mdt/performancedataqueryservlet
XSD schema of servlet response is accessible via URI:
/mdt/monitor/PerformanceDataQuery.xsd
Semantics of the constructed URI for querying servlet is:
/mdt/performancedataqueryservlet?component=<XIComponentName>&begin=<IntervalBeginTime>&end=<IntervalEndTime>
When querying the servlet, following parameters can be used:
| Parameter | Description | Value format / Possible values |
|---|---|---|
| component | Component of Advanced Adapter Engine. Components can be checked in Component Monitor or in SLD, in registered CIM instances of class SAP_XIAdapterFramework). Examples are central Adapter Engine and non-central Adapter Engine | |
| begin | Time period begin time stamp | YYYY-MM-DD hh:mm:ss.0 |
| end | Time period end time stamp | YYYY-MM-DD hh:mm:ss.0 |
If the servlet is called without parameters, the response will contain a list of components that can be queried.
If the servlet is called and a valid value for the component is passed, the response will contain a list of time periods per period type for which performance data is available for the specified component. Following period types are supported:
- MINUTE: performance data for last 1 hour, aggregated per 15 minutes;
- HOURLY: performance data for last 24 hours, aggregated per 1 hour;
- DAILY: performance data for last 7 days, aggregated per 1 day.
For PI releases 7.31/7.4, the enhancement was introduced that allows extension of retention period for persisted performance data. Using the parameter PeriodConfig of PerformanceDataQueryServlet, custom values for retention periods can be set. Please note that if Performance Monitor is deactivated, and then activated again, custom values for retention periods for persisted performance data will be overwritten with their default values given above. For more details, please read the SAP Note 2006366.
If the servlet is called and valid values for component and time period are passed, the response will contain performance data for processed messages – namely:
- Scenario attributes (scenario identifier, sender and receiver communication parties, components, communication channels, interface, direction, delivery semantics, etc.);
- Server node that processed messages for the scenario;
- Message counter;
- Important key figures reflecting performance data (maximum, minimum and average message size and retry attempts);
- Measuring points data (for each measuring point, maximum, minimum and average values in microseconds are provided).
Example request (URI part):
/mdt/performancedataqueryservlet?component=af.po1.nced60229921a&begin=2014-02-24 01:00:00.0&end=2014-02-25 01:00:00.0
Response (extract depicting a single performance data set entry):
Names of respective entry elements of data rows are provided at the beginning of the servlet response XML document:
The servlet usage is described in the SAP Note 1790113. The servlet is available as of PI release 7.31. Please consult the SAP Note for minimum SP/PL level requirements for the particular PI release.
Outro
You can also check the blog “Michal’s PI tips: How to get RWB message overview data to an external system?” written by Michal Krawczyk that discusses usage of MessageOverviewQueryServlet: Michal’s PI tips: How to get RWB message overview data to an external system? .
It should be noted that described servlets are not the only possibility to access message overview and performance data in Advanced Adapter Engine. On lower level, both servlets are calling ProfileProcessor – a component of Adapter Framework that is used for querying message statistics. Apart from discussed servlets, there are also several other tools that are working as proxies over ProfileProcessor – for example, the web service named ProfileProcessorVi. Consumer of this web service can retrieve message overview and performance data using web service operation getWSData() and passing corresponding application, profile and view keys together with time period and period type.
Overview table of message overview and performance applications as well as profiles and views that are supported by the web service is provided below:
| Retrieved data | Application key | Profile key | View key |
|---|---|---|---|
| Message overview data | sap – XPI_STATISTIC – http://sap.com/xi/XI/Message/30 | XPI |
R_ENTRY_VIEW_XPI S_ENTRY_VIEW_XPI SR_ENTRY_OVERVIEW_XPI SR_ENTRY_VIEW_XPI |
| Performance data | sap – XPI_PERFORMANCE_STATISTIC – http://sap.com/xi/XI/Message/30 | XPI_PERFORMANCE | ENTRY_VIEW_XPI_PERFORMANCE |
The web service ProfileProcessorVi is used by some SAP standard applications that are querying PI message statistics – for example, some applications of Solution Manager are actively using this web service. But consumption of this web service is more complicated in comparison to described servlets – just some reasons for this are listed below:
- More complex structure of the request that needs to be constructed and sent to the web service;
- More complex structure of the response that needs to be traversed (for example, actual statistics data is put in a hash form that requires further processing efforts);
- Additional operations that are required for correct completion of work with the ProfileProcessor after message statistics data is retrieved – namely, release of lock that have to be addressed in extra calls to the web service (by invocation of its operation processAction()).
Taking all these concerns into consideration, the web service ProfileProcessorVi can be considered as an alternative for servlets MessageOverviewQueryServlet and PerformanceDataQueryServlet, but is not really exposed officially for external usage in custom solutions and thus is not in scope of this blog.
hi Vadim,
maybe you can also add other blogs which were showing the same thing in the referecens for your blog:
Michal's PI tips: How to get RWB message overview data to an external system?
so we have all links to the same topic in new blogs,
Thanks,
Regards,
Michal Krawczyk
Hi Michal,
Thank you for this remark - this is surely a valid point, I added a reference to your blog here.
Regards,
Vadim
Dear Vadim,
Thanks for the post!
I'm wondering, if one changes the period DAILY for let's say 400 days, will there be an aggregation to periods weekly, monthly and annually?
And how long will these aggregated periods contained?
I did even use the use the servlet with mdt/performancedataqueryservlet?PeriodConfig=ANNUALLY=6 and it does return a OK result..
Do have any experience with a different configuration for performance monitoring in regard to available periods and automatic aggregation?
Regards,
Heiko
Hello Heiko,
Aggregation for different time buckets should be handled individually - thus,you should really specify specific values for different time buckets if you would like to reduce/increase them, otherwise the default value will be effective.
As for the functionality... We have tested it, but I personally do not have experience in setting time periods to relatively large values since it will have significant negative impact on performance of the PI system and should be set very carefully and it is not really recommended to keep performance data for too long time periods in PI unless there is any strong reason for this.
Regards,
Vadim
Hello Vadim,
we also would like to increase the captured interval, maybe even to include intervals for monthly and annually.
You are saying that performance wise it's not recommended to keep performance data for a long period of time. Could you elaborate on this a little? My guts feeling is, that this should not greatly impact operations as the data is aggregated only once and then kept as is. Even when keeping say 400 days with e.g. 60 scenarios and 2 server nodes we would end up with roughly 50.000 rows in the respective table. While this is not a small number, it's in my opinion no scale to be afraid of in a modern database environment.
Do you have any other concerns to keep this small?
Many thanks and kind regards
Jens
Hi Jens,
The reason behind recommendation of not keeping retention period for Adapter Engine's performance data too high is, as you correctly depicted, risk of increase of corresponding database tables' size (primarily, table XI_AF_PROF_PERF_AC). SAP Note 2006366 contains calculation that can be used to make rough estimation of number of rows in it, base on cluster size (number of server nodes) and number of interfaces / scenarios, combined with number of kept time buckets (for minute, daily and hourly aggregations). Note that for every period type (minute, daily, hourly), aggregated key figures are stored in separate table rows - hence, aggregated performance data for processed messages per server node and interface will occupy multiple table rows in the mentioned table.
I'm not aware of any specific threshold that defines optimal or maximum allowed size of this table, but the general rule for messages persistence layer (including those tables containing aggregated data - such as performance data), is to keep them as small and light as possible, unless you face specific and reasonable requirement and strongly need to increase messages / performance data retention period.
Regards,
Vadim
Thanks Vadim for this detailed answer.
One reason behind this is that we are running an 7.4 AEX with a per volume license. SAP does provide means of System measurement with note https://launchpad.support.sap.com/#/notes/2199183/E but at least in our SP 12 this doesn’t seem to work. Hence we are in the Need of externally Monitoring the consumed volume. Increasing this threshold would remedy the burden of capturing that data every x (Default would be 7) days.
As you seem to know your way around in this Topic, do you by Chance know if decreasing the bucket sizes (e.g. from 400 days to 30 days) will trigger Housekeeping of the affected tables? If that’s so we could at least easily revert to a working state, should a high bucket size greatly decrease Performance.
Many thanks and Kind regards
Jens
PS: Please forgive wrong capitalization. I’m forced to use IE with German Language Pack only currently as Chrome seems to have Trouble with my Client certificate and SSO in sap.com.
Hi Jens,
Yes, you are correct: there is a periodic job that checks if there are any persisted entries for message performance data, that exceed currently allowed number of periods for that specific period type (namely, minute, hourly and daily), and if so, deletes them from the database table.
On the other hand, if you need to keep performance data for further historic analysis, then you can look into option of keeping its retention period default / not too high, and make regular exports of this data to the external system / storage - either using built-in export to Excel functionality in Performance Monitor, or, to make it more automated, using PerformanceDataQueryServlet.
Given you tend to use this information for volume analysis in conjunction with licensing, please be aware that volume calculated for licensing purposes, might not come as a sum of all interfaces for which you find key figures in Performance Monitor - precisely, this is subject for attention when you differentiate between interfaces that integrate only SAP systems and those that involve at least one non-SAP system.
Regards,
Vadim
Thanks Vadim for pointing out that SAP to SAP traffic is free of Charge. We are Aware of this and separately Keep a list which identifies (as far as we think) those free of Charge Services.
I will take a more detailed look to PerformanceDataQueryServlet. It seems legit to use that for automatic Export.
I was trying to automate Excel Export from PIMON via some Screen scraping Tools like Wildfire but currently to no avail.
Many thanks again for the Input. Will come back here (at best with a link to an own blog) if I got my solution together or SAP provided a working measurement tool for AEX
Hi Vadim,
Very useful, complete and accurate post. Congrats.
I recently started using the MessageOverviewQueryServlet. In order to make it more usable and ABAP-proxable, I've created a simple PI synch flow which wraps it via SOAP (so it's SOAP->HTTP).
I've gone through the mentioned 1400785 SAP note which states that "The servlet is protected by basic HTTP authorization mechanism"
But if I try accessing it via Basic Auth, be that with PI or any http client like ARC or PostMan, I'm always presented the standard SAP J2EE auth form, i.e. the servlet is not called.
Do you have any idea why this happens or how I can tweak the auth chain of mdt?
(I'm on PI single stack 7.40 SPS13)
Thanks in advance
Alex
Hello Alex,
The servlet shall work well with basic authentication - can you please check in Postman that you provided valid credentials of the user with sufficient PI monitoring authorizations and updated the request with entered credentials ('Authorization' header is set in the request).
Regards,
Vadim
Hello Vadim,
Thanks for your kind reply.
Actually I am 100% sure I am using proper credentials since when I enter them in the infamous form, I get data. It looks like they're not enough via basic authentication...
This is the auth chain I can see (supposing it's the right web module...). What do you suggest? Shall I raise a ticket to SAP support?
Cheers
Alex
Hi Alex,
Can you please collect an HTTP trace on PO side when you make calls e.g. via Postman, and check details of the error context?
From configuration perspective, the screenshot you provided, follows the default setup - which is sufficient for basic authentication to be used when calling this servlet. And I have just checked this in an internal system (also 7.4, but lower SP level) - worked well for me when being tested from Postman with basic authentication.
Regards,
Vadim
I must be dumb...! 🙂
Thanks Vadim.
In fact it perfectly works via basic auth... Don't know how but it looks like I was using wrong credentials...
Thanks a lot for your help.
Alex
You are welcome, Alex. Good to know the problem has been fixed.
Regards,
Vadim
Hello Vadim
We tried to execute the below query for one day duration but it crashed the PI server nodes and even for 15 minutes it took time and while drilling down it is spinning. This query was working fine until last week but from yesterday onwards we are facing issue.
http://host:port/mdt/performancedataqueryservlet?component=af.aaa.hostdb&begin=2017-02-19 00:00:00.0&end=2017-02-20 00:00:00.0
In memory analysis this thread took more than 1.2GB memory. is there any workaround to extract this details
Regards
Bparthi
Hello Bparthi,
You can try reducing extraction time period for every call together with increasing periodicity of performance data queries - precisely, instead of calling it once a day for daily statistics, call it on hourly basis to extract hourly statistics.
Together with this, it would be helpful to see details of the obtained heap dump and analyze which objects contributed most to heap consumption. This will provide more specific outlook at what was potential cause (or causes) leading to extensive memory consumption.
Regards,
Vadim
Hello Vadim
We tried for 15 minutes which took more time to extract and while trying to drill down it was spinning. when we analyze the memory dump the below value shows as maximum memory acquired.
ef |value|\u0009<Data>\u000a\u0009\u0009<ColumnNames>\u000a\u0009\u0009\u0009<Column>INBOUND_CHANNEL</Column>\u000a\u0009\u0009\u0009<Column>OUTBOUND_CHANNEL</Column>\u000a\u0009\u0009\u0009<Column>DIRECTION</Column>\u000a\u0009\u0009\u0009<Column>DELIVERY_SEMANTICS</Column>\u000a\u0009\u0009\u0009<Column>SERVER_NODE</Column>\u000a\u0009\u0009\u0009<Column>FROM_PARTY_NAME</Column>\u000a\u0009\u0009\u0009<Column>FROM_SERVICE_NAME</Column>\u000a\u0009\u0009\u0009<Column>TO_PARTY_NAME</Column>\u000a\u0009\u0009\u0009<Column>TO_SERVICE_NAME</Column>\u000a\u0009\u0009\u0009<Column>ACTION_NAME</Column>\u000a\u0009\u0009\u0009<Column>ACTION_TYPE</Column>\u000a\u0009\u0009\u0009<Column>SCENARIO_IDENTIFIER</Column>\u000a\u0009\u0009\u0009<Column>MESSAGE_COUNTER</Column>\u000a\u0009\u0009\u0009<Column>MAX_MESSAGE_SIZE</Column>\u000a\u0009\u0009\u0009<Column>MIN_MESSAGE_SIZE</Column>\u000a\u0009\u0009\u0009<Column>AVG_MESSAGE_SIZE</Column>\u000a\u0009\u0009\u0009<Column>MAX_RETRY_COUNTER</Column>\u000a\u0009\u0009\u0009<Column>MIN_RETRY_COUNTER</Column>\u000a\u0009\u0009\u0009<Column>AVG_RETRY_COUNTER</Column>\u000a\u0009\u0009\u0009<Column>MEASURING_POINTS</Column>\u000a\u0009\u0009\u0009<Column>AVG_PROCESSING_TIME</Column>\u000a\u0009\u0009\u0009<Column>TOTAL_PROCESSING_TIME</Column>\u000a\u0009\u0009\u0009<Column>MEASURING_POINTS</Column>\u000a\u0009\u0009\u0009<Column>AVG_PROCESSING_TIME</Column>\u000a\u0009\u0009\u0009<Column>TOTAL_PROCESSING_TIME</Column>\u000a\u0009\u0009\u0009<Column>MEASURING_POINTS</Column>\u000a\u0009\u0009\u0009<Column>AVG_PROCESSING_TIME</Column>\u000a\u0009\u0009.
Well, it is not really something I was looking for - from the heap dump, it is essential to investigate which objects were largest and also which references to other objects could cause high heap consumption (assuming you can already narrow down analysis by paying special attention to those objects related to Performance Monitor and its underlying functionality). This will make further statements in analysis more specific and focused on potential causes of the problem.
Regards,
Vadim
We have been using the performance data servlet to programmatically get the data and write it to a SQL Server database so we can have long term performance statistics without the performance impact of keeping a lot of data in PI. This had been working ok until we applied on patch and then we basically had the same issue, as per SAP response to our OSS note, solving the issue would need installing several patches so since we had already planned to upgrade our PI servers we are upgrading to 7.31 SP20 and we have confirmed that the servlet works as before after the upgrade. So check your versions and you may want to create an OSS note to obtain the solution appropriate to your environment.
Hi all,
I can access the message overview from a SOAPUI with this described method, OK, well done!!! but... I need to access the same service in a 7.0 old XI and an error occurs ( NW04S_19_REL )
The requested resource does not exist.
May be in old systems this service are renamed ?
Hello Juan,
I would suggest cross-checking Adapter Framework (SAP_XIAF) software component's version installed in your PI 7.0 system against those where MessageOverviewQueryServlet is available for, which can be done going through the SAP Note 1400785. In particular, you might see from that SAP Note, that it doesn't state availability of the servlet in question, in SP19 of 7.0 (but only in limited number of Support Packages for release 7.0).
Regards,
Vadim
Hello Vadim, Nice and useful blog.
Do you know how to do this kind of monitorization for bpm? Is there an url such as /mdt/messageoverviewqueryservlet?
Thanks.
Hello Vadim,
I am trying to get SAP PO message statistics from an Angularjs(Angular 4) application. For this I am making a Get request to messageoverviewqueryservlet as given in this blog
I am expecting a response similar to the one given below
but what I get seems to be the HTML code for login page. Also whenever I enter the link
(http://<PO_server>:<PO_port>/mdt/messageoverviewqueryservlet?component=<XIComponentName>&view=<ViewNameKey>&begin=<IntervalBeginTime>&end=<IntervalEndTime>) in the browser I get stuck at the login window and get the response only after I login.
Is there a way to pass username and password to this page from a program so that I can get the XML response in my Angular app? I have tried passing username and password via get request headers but that didn't work
Can anyone please help me solve this?
Thanks,
Shardul Sane
Hello Shardul,
As the most basic option, basic authentication can be an option to go - which means, usage of basic authentication method / setting 'Authorization' header with the value 'Basic ' followed by user/password encoded value, when issuing GET request to the queried servlet. More advanced options (involving authentication methods other than basic) will depend on what other authentication methods are configured in your SAP PO system.
Regards,
Vadim
Hello Vadim,
Great blog!!!. thanks you
I am trying to get SAP PO message overview with component local Integration Engines (proxy runtimes of connected application systems), but the service response with a code result
APPLICATION_ERROR
CentralViewManager.getIProfiles()
Unable to load profiles for the Message Overview application from component Proxy Runtime
Please, Could you help me?
Thanks you very much
Fede
Hi Fede,
I know it's late to reply - but i just faced the same issue and did not found much literature on this... Hence posting the way i overcame it here. Don't use the abap backend (proxy) as component. Use the af.* as component instead: you'll see everything from here.
Thanks Vadim for the great blog.
Hi Jean
Thanks you. Could you share the link please?
Thanks you!!!
Regards
Fede
hi Fede,
Sorry i wasn't clear - i've found nothing at all about this issue, except your post, hence i'm writing here. A colleague of mine (PO expert) helped me on this, i'll share his initials for the record: B.B 🙂
cheers, J
Hi Jean - Could you please provide some more details on using the af.* as component? I am currently getting this error - Unable to load profiles for the Message Overview application from component Proxy Runtime.
Hi Vijay,
Sorry for my late reply - indeed i should have shared this before: the Adapter Framework component name, beginning with af.<SID>.<ip_or_name>, is grabbed from /mdt/messageoverviewqueryservlet url path.
cheers, jean
Hello Federico - Were you able to find a solution to this issue? I am facing the same problem and getting the error message - Unable to load profiles for the Message Overview application from component Proxy Runtime
Hi Vadim:
thanks for your great job!
I found an insufficiency that PerformanceDataQueryServlet can't get total message size, just MAX/MIN/AVG only.
Why? Do you have some idea to get total message size data?
Thanks again.
Ted.