Skip to Content

Intro

SAP PI system is already shipped with extensive toolset of monitoring applications that can provide aggregated statistics on messages that have been processed in Advanced Adapter Engine (from message overview and performance perspectives). In addition to built-in tools, the one can employ functionality of Solution Manager for centralized monitoring of the PI system in general and statistics of message processing in particular. But there are some cases, when these tools are not sufficient and it is necessary to retrieve statistics of message processing in Advanced Adapter Engine of the SAP PI system for external usage – e.g. in case the consumer of this data is the external monitoring application.

 

To achieve this requirement, it is possible to query two standard servlets of Advanced Adapter Engine that can provide necessary information via HTTP (thus, accessing PI host via J2EE HTTP port – by default, 5<instance number>00). In this blog, I will provide an outlook at their functionality and use cases. The discussed servlets are:

  • MessageOverviewQueryServlet
  • PerformanceDataQueryServlet

 

Both servlets are exposed by Message Display Tool application of Advanced Adapter Engine:

 

/wp-content/uploads/2014/02/servlets_395718.png

 

Requests for these servlets are constructed using input parameters (passed as GET parameters in the URL) and responses are rendered in a form of XML document that can be traversed and analyzed conveniently by the caller application. XSD schemas of responses of the mentioned servlets are accessible via URI /mdt/monitor/<servlet name>.xsd.

 

 

MessageOverviewQueryServlet

The servlet is used to access message processing overview data externally. In PIMON, this is data displayed in Message Monitor -> Message Status Overview.

 

The servlet is accessible via URI:

/mdt/messageoverviewqueryservlet

 

XSD schema of servlet response is accessible via URI:

/mdt/monitor/MessageOverviewQuery.xsd

 

Semantics of the constructed URI for querying servlet is:

/mdt/messageoverviewqueryservlet?component=<XIComponentName>&view=<ViewNameKey>&begin=<IntervalBeginTime>&end=<IntervalEndTime>[&detailedStatus=<true|false>]

 

When querying the servlet, following parameters can be used:

 

Parameter Description Value format / Possible values
component PI component. Components can be checked in Component Monitor. Examples are central Adapter Engine and non-central Adapter Engine, central Integration Engine and local Integration Engines (proxy runtimes of connected application systems)
view View key. The view defines a set of retrieved attributes and sorting principles for queried message overview statistics

R_ENTRY_VIEW_XPI

S_ENTRY_VIEW_XPI

SR_ENTRY_OVERVIEW_XPI

SR_ENTRY_VIEW_XPI

begin Time period begin time stamp YYYY-MM-DD hh:mm:ss.0
end Time period end time stamp YYYY-MM-DD hh:mm:ss.0
detailedStatus Indicator for showing status groups / detailed status values

true

false

 

If the servlet is called without parameters, the response will contain a list of components that can be queried.

If the servlet is called and a valid value for the component is passed, the response will contain a list of view keys and time periods per period type for which message overview data is available for the specified component. Following period types are supported:

  • HOURLY: message overview data for last 12 hours, aggregated per 1 hour;
  • DAILY: message overview data for last 7 days, aggregated per 1 day;
  • WEEKLY: message overview data for last month, aggregated per 1 week;
  • MONTHLY: message overview data for last 12 months, aggregated per 1 month;
  • ANNUALLY: message overview data for several years, aggregated per 1 year.

 

If the servlet is called and valid values for component, view and time period are passed, the response will contain message overview data for processed messages – namely:

  • Scenario attributes (scenario identifier, sender and receiver communication components, interface);
  • Message counter per message status (for example, for statuses ‘System error’, ‘Waiting’, ‘To be delivered’, ‘Delivering’, ‘Holding’, ‘Successful’, ‘Canceled with errors’ for Advanced Adapter Engine).

 

Example request (URI part):

/mdt/messageoverviewqueryservlet?component=af.po1.nced60229921a&view=SR_ENTRY_VIEW_XPI&begin=2014-02-24 01:00:00.0&end=2014-02-25 01:00:00.0&detailedStatus=true

 

Response (extract depicting two message overview data set entries):

 

message overview - response.png

 

Names of respective entry elements of data rows are provided at the beginning of the servlet response XML document:

 

message overview - columns.png

 

The servlet usage is described in the SAP Note 1400785. The servlet is available as of PI release 7.0. Please consult the SAP Note for minimum SP/PL level requirements for the particular PI release.

 

 

PerformanceDataQueryServlet

The servlet is used to access message processing performance data externally. In PIMON, this is data displayed in Performance Monitor.

 

The servlet is accessible via URI:

/mdt/performancedataqueryservlet

 

XSD schema of servlet response is accessible via URI:

/mdt/monitor/PerformanceDataQuery.xsd

 

Semantics of the constructed URI for querying servlet is:

/mdt/performancedataqueryservlet?component=<XIComponentName>&begin=<IntervalBeginTime>&end=<IntervalEndTime>

 

When querying the servlet, following parameters can be used:

 

Parameter Description Value format / Possible values
component Component of Advanced Adapter Engine. Components can be checked in Component Monitor or in SLD, in registered CIM instances of class SAP_XIAdapterFramework). Examples are central Adapter Engine and non-central Adapter Engine
begin Time period begin time stamp YYYY-MM-DD hh:mm:ss.0
end Time period end time stamp YYYY-MM-DD hh:mm:ss.0

 

If the servlet is called without parameters, the response will contain a list of components that can be queried.

If the servlet is called and a valid value for the component is passed, the response will contain a list of time periods per period type for which performance data is available for the specified component. Following period types are supported:

  • MINUTE: performance data for last 1 hour, aggregated per 15 minutes;
  • HOURLY: performance data for last 24 hours, aggregated per 1 hour;
  • DAILY: performance data for last 7 days, aggregated per 1 day.

 

For PI releases 7.31/7.4, the enhancement was introduced that allows extension of retention period for persisted performance data. Using the parameter PeriodConfig of PerformanceDataQueryServlet, custom values for retention periods can be set. Please note that if Performance Monitor is deactivated, and then activated again, custom values for retention periods for persisted performance data will be overwritten with their default values given above. For more details, please read the SAP Note 2006366.

 

If the servlet is called and valid values for component and time period are passed, the response will contain performance data for processed messages – namely:

  • Scenario attributes (scenario identifier, sender and receiver communication parties, components, communication channels, interface, direction, delivery semantics, etc.);
  • Server node that processed messages for the scenario;
  • Message counter;
  • Important key figures reflecting performance data (maximum, minimum and average message size and retry attempts);
  • Measuring points data (for each measuring point, maximum, minimum and average values in microseconds are provided).

 

Example request (URI part):

/mdt/performancedataqueryservlet?component=af.po1.nced60229921a&begin=2014-02-24 01:00:00.0&end=2014-02-25 01:00:00.0

 

Response (extract depicting a single performance data set entry):

 

performance - response.png

 

Names of respective entry elements of data rows are provided at the beginning of the servlet response XML document:

 

performance - columns.png

 

The servlet usage is described in the SAP Note 1790113. The servlet is available as of PI release 7.31. Please consult the SAP Note for minimum SP/PL level requirements for the particular PI release.

 

 

Outro

You can also check the blog “Michal’s PI tips: How to get RWB message overview data to an external system?” written by Michal Krawczyk that discusses usage of MessageOverviewQueryServlet: Michal’s PI tips: How to get RWB message overview data to an external system? .

 

It should be noted that described servlets are not the only possibility to access message overview and performance data in Advanced Adapter Engine. On lower level, both servlets are calling ProfileProcessor – a component of Adapter Framework that is used for querying message statistics. Apart from discussed servlets, there are also several other tools that are working as proxies over ProfileProcessor – for example, the web service named ProfileProcessorVi. Consumer of this web service can retrieve message overview and performance data using web service operation getWSData() and passing corresponding application, profile and view keys together with time period and period type.

 

WS Navigator - service.png

 

WS Navigator - operations.png

 

Overview table of message overview and performance applications as well as profiles and views that are supported by the web service is provided below:

 

Retrieved data Application key Profile key View key
Message overview data sap – XPI_STATISTIC – http://sap.com/xi/XI/Message/30 XPI

R_ENTRY_VIEW_XPI

S_ENTRY_VIEW_XPI

SR_ENTRY_OVERVIEW_XPI

SR_ENTRY_VIEW_XPI

Performance data sap – XPI_PERFORMANCE_STATISTIC – http://sap.com/xi/XI/Message/30 XPI_PERFORMANCE ENTRY_VIEW_XPI_PERFORMANCE

 

The web service ProfileProcessorVi is used by some SAP standard applications that are querying PI message statistics – for example, some applications of Solution Manager are actively using this web service. But consumption of this web service is more complicated in comparison to described servlets – just some reasons for this are listed below:

  • More complex structure of the request that needs to be constructed and sent to the web service;
  • More complex structure of the response that needs to be traversed (for example, actual statistics data is put in a hash form that requires further processing efforts);
  • Additional operations that are required for correct completion of work with the ProfileProcessor after message statistics data is retrieved – namely, release of lock that have to be addressed in extra calls to the web service (by invocation of its operation processAction()).

 

Taking all these concerns into consideration, the web service ProfileProcessorVi can be considered as an alternative for servlets MessageOverviewQueryServlet and PerformanceDataQueryServlet, but is not really exposed officially for external usage in custom solutions and thus is not in scope of this blog.

To report this post you need to login first.

24 Comments

You must be Logged on to comment or reply to a post.

  1. Heiko Bergmann

    Dear Vadim,

    Thanks for the post!

    I’m wondering, if one changes the period DAILY for let’s say 400 days, will there be an aggregation to periods weekly, monthly and annually?

    And how long will these aggregated periods contained?


    I did even use the use the servlet with mdt/performancedataqueryservlet?PeriodConfig=ANNUALLY=6  and it does return a OK result..

    Do have any experience with a different configuration for performance monitoring in regard to available periods and automatic aggregation?

    Regards,

    Heiko

    (0) 
    1. Vadim Klimov Post author

      Hello Heiko,

      Aggregation for different time buckets should be handled individually – thus,you should really specify specific values for different time buckets if you would like to reduce/increase them, otherwise the default value will be effective.

      As for the functionality… We have tested it, but I personally do not have experience in setting time periods to relatively large values since it will have significant negative impact on performance of the PI system and should be set very carefully and it is not really recommended to keep performance data for too long time periods in PI unless there is any strong reason for this.

      Regards,

      Vadim

      (0) 
      1. Jens Schwendemann

        Hello Vadim,

        we also would like to increase the captured interval, maybe even to include intervals for monthly and annually.

        You are saying that performance wise it’s not recommended to keep performance data for a long period of time. Could you elaborate on this a little? My guts feeling is, that this should not greatly impact operations as the data is aggregated only once and then kept as is. Even when keeping say 400 days with e.g. 60 scenarios and 2 server nodes we would end up with roughly 50.000 rows in the respective table. While this is not a small number, it’s in my opinion no scale to be afraid of in a modern database environment.

        Do you have any other concerns to keep this small?

         

        Many thanks and kind regards

        Jens

        (0) 
        1. Vadim Klimov Post author

          Hi Jens,

          The reason behind recommendation of not keeping retention period for Adapter Engine’s performance data too high is, as you correctly depicted, risk of increase of corresponding database tables’ size (primarily, table XI_AF_PROF_PERF_AC). SAP Note 2006366 contains calculation that can be used to make rough estimation of number of rows in it, base on cluster size (number of server nodes) and number of interfaces / scenarios, combined with number of kept time buckets (for minute, daily and hourly aggregations). Note that for every period type (minute, daily, hourly), aggregated key figures are stored in separate table rows – hence, aggregated performance data for processed messages per server node and interface will occupy multiple table rows in the mentioned table.

          I’m not aware of any specific threshold that defines optimal or maximum allowed size of this table, but the general rule for messages persistence layer (including those tables containing aggregated data – such as performance data), is to keep them as small and light as possible, unless you face specific and reasonable requirement and strongly need to increase messages / performance data retention period.

          Regards,

          Vadim

          (1) 
          1. Jens Schwendemann

             

            Thanks Vadim for this detailed answer.

             

            One reason behind this is that we are running an 7.4 AEX with a per volume license. SAP does provide means of System measurement with note https://launchpad.support.sap.com/#/notes/2199183/E but at least in our SP 12 this doesn’t seem to work. Hence we are in the Need of externally Monitoring the consumed volume. Increasing this threshold would remedy the burden of capturing that data every x (Default would be 7) days.

             

            As you seem to know your way around in this Topic, do you by Chance know if decreasing the bucket sizes (e.g. from 400 days to 30 days) will trigger Housekeeping of the affected tables? If that’s so we could at least easily revert to a working state, should a high bucket size greatly decrease Performance.

            Many thanks and Kind regards

            Jens

            PS: Please forgive wrong capitalization. I’m forced to use IE with German Language Pack only currently as Chrome seems to have Trouble with my Client certificate and SSO in sap.com.

             

             

            (0) 
            1. Vadim Klimov Post author

              Hi Jens,

              Yes, you are correct: there is a periodic job that checks if there are any persisted entries for message performance data, that exceed currently allowed number of periods for that specific period type (namely, minute, hourly and daily), and if so, deletes them from the database table.

              On the other hand, if you need to keep performance data for further historic analysis, then you can look into option of keeping its retention period default / not too high, and make regular exports of this data to the external system / storage – either using built-in export to Excel functionality in Performance Monitor, or, to make it more automated, using PerformanceDataQueryServlet.

              Given you tend to use this information for volume analysis in conjunction with licensing, please be aware that volume calculated for licensing purposes, might not come as a sum of all interfaces for which you find key figures in Performance Monitor – precisely, this is subject for attention when you differentiate between interfaces that integrate only SAP systems and those that involve at least one non-SAP system.

              Regards,

              Vadim

              (1) 
              1. Jens Schwendemann

                 

                Thanks Vadim for pointing out that SAP to SAP traffic is free of Charge. We are Aware of this and separately Keep a list which identifies (as far as we think) those free of Charge Services.

                 

                I will take a more detailed look to PerformanceDataQueryServlet. It seems legit to use that for automatic Export.

                I was trying to automate Excel Export from PIMON via some Screen scraping Tools like Wildfire but currently to no avail.

                 

                Many thanks again for the Input. Will come back here (at best with a link to an own blog) if I got my solution together or SAP provided a working measurement tool for AEX 🙂

                (0) 
  2. Alessandro Guarneri

    Hi Vadim,
    Very useful, complete and accurate post. Congrats.

    I recently started using the MessageOverviewQueryServlet. In order to make it more usable and ABAP-proxable, I’ve created a simple PI synch flow which wraps it via SOAP (so it’s SOAP->HTTP).
    I’ve gone through the mentioned 1400785 SAP note which states that “The servlet is protected by basic HTTP authorization mechanism”

    But if I try accessing it via Basic Auth, be that with PI or any http client like ARC or PostMan, I’m always presented the standard SAP J2EE auth form, i.e. the servlet is not called.
    Do you have any idea why this happens or how I can tweak the auth chain of mdt?
    (I’m on PI single stack 7.40 SPS13)

    Thanks in advance

    Alex

    (0) 
    1. Vadim Klimov Post author

      Hello Alex,

      The servlet shall work well with basic authentication – can you please check in Postman that you provided valid credentials of the user with sufficient PI monitoring authorizations and updated the request with entered credentials (‘Authorization’ header is set in the request).

      Regards,
      Vadim

      (0) 
      1. Alessandro Guarneri

        Hello Vadim,
        Thanks for your kind reply.
        Actually I am 100% sure I am using proper credentials since when I enter them in the infamous form, I get data. It looks like they’re not enough via basic authentication…
        This is the auth chain I can see (supposing it’s the right web module…). What do you suggest? Shall I raise a ticket to SAP support?
        Cheers
        Alex

        (0) 
        1. Vadim Klimov Post author

          Hi Alex,

          Can you please collect an HTTP trace on PO side when you make calls e.g. via Postman, and check details of the error context?

          From configuration perspective, the screenshot you provided, follows the default setup – which is sufficient for basic authentication to be used when calling this servlet. And I have just checked this in an internal system (also 7.4, but lower SP level) – worked well for me when being tested from Postman with basic authentication.

          Regards,
          Vadim

          (0) 
          1. Alessandro Guarneri

            I must be dumb…! 🙂
            Thanks Vadim.
            In fact it perfectly works via basic auth… Don’t know how but it looks like I was using wrong credentials…
            Thanks a lot for your help.

            Alex

            (0) 
  3. bala t

    Hello Vadim
    We tried to execute the below query for one day duration but it crashed the PI server nodes and even for 15 minutes it took time and while drilling down it is spinning. This query was working fine until last week but from yesterday onwards we are facing issue.
    http://host:port/mdt/performancedataqueryservlet?component=af.aaa.hostdb&begin=2017-02-19 00:00:00.0&end=2017-02-20 00:00:00.0
    In memory analysis this thread took more than 1.2GB memory. is there any workaround to extract this details
    Regards
    Bparthi

    (0) 
    1. Vadim Klimov Post author

      Hello Bparthi,

      You can try reducing extraction time period for every call together with increasing periodicity of performance data queries – precisely, instead of calling it once a day for daily statistics, call it on hourly basis to extract hourly statistics.

      Together with this, it would be helpful to see details of the obtained heap dump and analyze which objects contributed most to heap consumption. This will provide more specific outlook at what was potential cause (or causes) leading to extensive memory consumption.

      Regards,

      Vadim

      (0) 
      1. bala t

        Hello Vadim

        We tried for 15 minutes which took more time to extract and while trying to drill down it was spinning. when we analyze the memory dump the below value shows as maximum memory acquired.

        ef |value|\u0009<Data>\u000a\u0009\u0009<ColumnNames>\u000a\u0009\u0009\u0009<Column>INBOUND_CHANNEL</Column>\u000a\u0009\u0009\u0009<Column>OUTBOUND_CHANNEL</Column>\u000a\u0009\u0009\u0009<Column>DIRECTION</Column>\u000a\u0009\u0009\u0009<Column>DELIVERY_SEMANTICS</Column>\u000a\u0009\u0009\u0009<Column>SERVER_NODE</Column>\u000a\u0009\u0009\u0009<Column>FROM_PARTY_NAME</Column>\u000a\u0009\u0009\u0009<Column>FROM_SERVICE_NAME</Column>\u000a\u0009\u0009\u0009<Column>TO_PARTY_NAME</Column>\u000a\u0009\u0009\u0009<Column>TO_SERVICE_NAME</Column>\u000a\u0009\u0009\u0009<Column>ACTION_NAME</Column>\u000a\u0009\u0009\u0009<Column>ACTION_TYPE</Column>\u000a\u0009\u0009\u0009<Column>SCENARIO_IDENTIFIER</Column>\u000a\u0009\u0009\u0009<Column>MESSAGE_COUNTER</Column>\u000a\u0009\u0009\u0009<Column>MAX_MESSAGE_SIZE</Column>\u000a\u0009\u0009\u0009<Column>MIN_MESSAGE_SIZE</Column>\u000a\u0009\u0009\u0009<Column>AVG_MESSAGE_SIZE</Column>\u000a\u0009\u0009\u0009<Column>MAX_RETRY_COUNTER</Column>\u000a\u0009\u0009\u0009<Column>MIN_RETRY_COUNTER</Column>\u000a\u0009\u0009\u0009<Column>AVG_RETRY_COUNTER</Column>\u000a\u0009\u0009\u0009<Column>MEASURING_POINTS</Column>\u000a\u0009\u0009\u0009<Column>AVG_PROCESSING_TIME</Column>\u000a\u0009\u0009\u0009<Column>TOTAL_PROCESSING_TIME</Column>\u000a\u0009\u0009\u0009<Column>MEASURING_POINTS</Column>\u000a\u0009\u0009\u0009<Column>AVG_PROCESSING_TIME</Column>\u000a\u0009\u0009\u0009<Column>TOTAL_PROCESSING_TIME</Column>\u000a\u0009\u0009\u0009<Column>MEASURING_POINTS</Column>\u000a\u0009\u0009\u0009<Column>AVG_PROCESSING_TIME</Column>\u000a\u0009\u0009.

        (0) 
        1. Vadim Klimov Post author

          Well, it is not really something I was looking for – from the heap dump, it is essential to investigate which objects were largest and also which references to other objects could cause high heap consumption (assuming you can already narrow down analysis by paying special attention to those objects related to Performance Monitor and its underlying functionality). This will make further statements in analysis more specific and focused on potential causes of the problem.

          Regards,

          Vadim

          (0) 
  4. Juan Pascuet

    Hi all,

    I can access the message overview from a SOAPUI with this described method, OK, well done!!! but… I need to access the same service in a 7.0 old XI and an error occurs ( NW04S_19_REL )

      The requested resource does not exist.

     

    May be in old systems this service are renamed ?

     

     

    (0) 
    1. Vadim Klimov Post author

      Hello Juan,

       

      I would suggest cross-checking Adapter Framework (SAP_XIAF) software component’s version installed in your PI 7.0 system against those where MessageOverviewQueryServlet is available for, which can be done going through the SAP Note 1400785. In particular, you might see from that SAP Note, that it doesn’t state availability of the servlet in question, in SP19 of 7.0 (but only in limited number of Support Packages for release 7.0).

       

      Regards,

      Vadim

      (0) 
  5. Shardul Sane

    Hello Vadim,

    I am trying to get SAP PO message statistics from an Angularjs(Angular 4) application. For this I am making a Get request to messageoverviewqueryservlet as given in this blog

    I am expecting a response similar to the one given below

    but what I get seems to be the HTML code for login page. Also whenever I enter the link

    (http://<PO_server&gt;:<PO_port>/mdt/messageoverviewqueryservlet?component=<XIComponentName>&view=<ViewNameKey>&begin=<IntervalBeginTime>&end=<IntervalEndTime>) in the browser I get stuck at the login window and get the response only after I login.

    Is there a way to pass username and password to this page from a program so that I can get the XML response in my Angular app? I have tried passing username and password via get request headers but that didn’t work

    Can anyone please help me solve this?

    Thanks,

    Shardul Sane

    (0) 
    1. Vadim Klimov Post author

      Hello Shardul,

      As the most basic option, basic authentication can be an option to go – which means, usage of basic authentication method / setting ‘Authorization’ header with the value ‘Basic ‘ followed by user/password encoded value, when issuing GET request to the queried servlet. More advanced options (involving authentication methods other than basic) will depend on what other authentication methods are configured in your SAP PO system.

      Regards,

      Vadim

      (0) 

Leave a Reply