ST05: Activate Trace Recording with Filter

[ManfredMensch]

This is the second blog post in my series of posts about SAP GUI transaction ST05. Here, I explain that you can control and customize the trace recording with filter conditions, and how this is useful for analyzing your applications’ performance. With a small case study I show this feature in action.

In the introduction to the sequence of blog posts (Use ST05 to Analyze the Communication of the ABAP Work Process with External Resources) I have referred to my transactions STATS and STATS_FE as measurement tools, which help you to find performance hot spots in your applications. To identify root causes of poor performance (long response time or high resource consumption), you need dedicated analysis tools. With respect to communications and data transfers between the ABAP work process and external components, ST05 is the analysis tool of choice. I have described the straightforward way of working with it in my previous post ST05: Basic Use. Be sure to read it for a quick review.

The simplest way to switch on tracing involves the selection of the desired trace types, followed by a click on Activate Trace. Then trace recording is active only on the local ABAP application server instance and exclusively for the current user. Activities on other instances or by other users are not captured.
Now imagine that you want to support co-workers with the performance analysis of their business application. Maybe they do not have authorization to run ST05, while you are not allowed to work with the application. Or you do not know ahead of time which of the system's application server instances will handle the back-end portion of the application—the system’s load balancer may dispatch incoming requests to any instance. Or you need to investigate a batch job that runs at night time, when you do not want to manually switch on or off the trace recording. Furthermore, within a trace type, i.e., for a particular communication interface of the ABAP work process, you might be interested only in a subset of all associated communication events. All these situations are not covered by the ordinary Activate Trace. This is where the option to Activate Trace with Filter (Fig. 1) comes in.
Figure 1: Clicking button Activate Trace with Filter opens a  popup with Conditions for Trace Recording. There you can override some of ST05’s defaults, or specify restrictions on the communication events to be captured. Scheduling the trace recording is convenient for the analysis of batch jobs that run while you are not in the office.
The running text explains all options in detail.

 

With an entry in field User Name you can activate tracing for another user. Then her or his activities are recorded instead of yours. User name patterns with the asterisk * as a terminating wildcard character are supported.

To trace events originating from or handled by a non-local application server instance within the current system, expand the dropdown list box for the Instance field to see all active instances. The local instance is explicitly indicated. If you know the relevant instance, select it. Otherwise, to activate tracing on all instances, choose <ALL>. This tries to switch on tracing, but if it is already active for another user on an instance, the attempt fails for this instance. A popup will inform about this failure. The brute force option <ENFORCE> unconditionally terminates any tracing in progress on all instances and then activates trace recording per your conditions. Use this option sparingly and carefully. It is convenient and safe for you, but may be disturbing and annoying for your colleagues.

With the group of fields Transaction Name, Program Name, RFC Function Name, or URL, you can limit the captured events to those that are triggered while executing a particular application. Note that this application must be at the very top of the call hierarchy. Therefore, a value can be given for at most one of the fields. Patterns with the asterisk * as a terminating wildcard are allowed.

Occasionally, an application (most often a batch job) may occupy a work process for a very long time. With transaction SM50 you can determine the work process’ number and then enter it into the Process Number field of the filter conditions popup. Thereby, you confine the trace recording to events related to the given ABAP work process. The resulting trace may identify the reason for the long duration. There is one caveat: A long running database operation that started before the trace was switched on will not trigger the creation of an associated trace record.
(To the best of my knowledge, the value of this filter criterion is rather limited, and it is not used very often. Please add a comment to this blog post if you know of an important use case.)

For an SQL trace, you can restrict the recording of SQL statements to those that work on certain database tables (or database views, including CDS views). You may define up to 5 tables to be included. Similarly, you can exclude up to 5 database tables. Statements accessing these excluded tables will then not be recorded. For both options, you may filter on groups of tables by terminating entries in columns Include or Exclude with an asterisk *. For implemented Open SQL JOINs, the table name filter applies to the first of the JOINed database tables. If a database table matches entries in both columns, Include and Exclude, SQL statements against this table will not be recorded.
If you specify a table in the Include column, but also want to trace all statements that do not directly affect an individual table (e.g., COMMITs, ROLLBACKs, SET SESSION VARIABLE, SET CLIENT INFO), you must mark the check box Include Statements with Empty Table Names.
You may identify tables to be included with transaction STATS(cf. my blog post Measure your Application’s Performance and Resource Consumption: Evaluate Statistics Records with Tr...): The Top N Table Accesses sub record in the DB area shows the top N most expensive tables. You may also decide to focus on a table with central importance for your line of business, so that you get an overview and deeper understanding of where or how this table is used. Based on this knowledge, you can derive approaches for optimizing the accesses to this table.
Excluding certain tables from trace recording may be helpful if your application triggers many statements against these tables and you have already verified and confirmed that they are all necessary and processed in the best possible way. Then the corresponding trace records would only clutter up your trace, without adding any useful information. Even worse, they increase the risk that the available trace files get filled up to their capacity and then are successively overwritten, thereby evicting potentially important records. (Refer to my blog post ST05: Technical Background of Trace Recording and Analysis for a detailed description and explanation of the underlying mechanism.)
Make sure that you do not by mistake exclude a table that your application accesses via potentially expensive statements.

Imagine that you need to investigate the communications triggered by a batch job. You know its scheduled start time and have an idea about its expected duration. If the job starts or ends well outside of your work time, you can use ST05’s Scheduling capability. Mark the check box Schedule Trace Recording, specify the relevant Start Date and Start Time as well as End Date and End Time. This schedules and releases jobs named ST05_Trace_On and ST05_Trace_Off, which respectively activate and deactivate tracing on the appropriate dates and times. Start Date and Start Time should be slightly earlier than the beginning of the batch job you want to analyze, and End Date and End Time should be a little later than its projected end. Only then will the trace recording be active for the job's entire duration so that the resulting trace can cover all relevant events.
If you cannot analyze the trace shortly after it was recorded, other subsequent tracing activities may overwrite it from its beginning. To protect against this, mark the check box Save Trace in DB and provide a Description. This will add a second step to job ST05_Trace_Off, which processes all available trace records in the prescribed time frame and persists them with your description into the trace directory. There they will expire only after 120 days, giving you plenty of time for your evaluation. Please note that an active trace recording may overwrite its own beginning. The danger  for this increases with the time during which tracing is switched on. Option Save Trace in DB cannot prevent this. It will nevertheless store the remaining trace records into the trace directory so that you can work at least with those that are still available, and do not lose even more records to other tracing activities. Of course, the previously explained approach to limit traced SQL statements to those that do or do not access specified database tables is very efficient in reducing the risk that a scheduled trace overwrites its own leading records.

Once you have defined your filter conditions, switch on trace recording by pressing the Enter key. The Trace State area on ST05’s main screen will then indicate that recording has been activated with filter. To review the current settings, select Display Filter for Trace Recording from the Edit menu (Fig. 2). If your only restriction affects the User Name, ST05 does not consider this a genuine filtered activation, but of course still respects it, and already displays it in the Trace State.
The User Name must not be changed while recording a trace. The Transaction Name, Program Name, RFC Function Name, or URL to be traced, as well as the list of tables to be Included or Excluded, can be modified while tracing.
If you Schedule Trace Recording, a message in the SAP GUI’s status bar indicates the result. On success the corresponding batch jobs show up in the job overview of transaction SM37.
Figure 2: Review the Conditions for Trace Recording, either in the Trace State area on ST05's main screen, or in full detail via a dedicated popup.

 

When you have switched on trace recording on all application server instances, make sure that you click button Deactivate Trace on All Instances to switch it off. Otherwise, it gets deactivated only on the local instance, but keeps going on all remote instances.

 

Example

Let's assume that we want to analyze the start of the SAP S/4HANA Fiori application F0997 (Audit Journal) with an automatically applied default filter on Company Code 1010, Fiscal Year 2023, and Ledger 0L. We start the app by selecting it from our favorites in the SAP S/4HANA Product Home Page, and then immediately go back after the list of journal entries has been displayed. We repeat this back and forth pattern several times to create a reliable set of statistics records. 
We begin our investigation with transaction STATS_FE(Use Transaction STATS_FE to Analyze the Performance of your Web Application) to measure the end-to-end performance of this front-end user interaction and to understand its overall impact on the ABAP back end (Fig. 3). We identify one front-end statistics record as representative for the user's activity. Forward navigation by double click takes us from there to transaction STATS, which resolves the user interaction's total effect into individual back-end tasks and their corresponding main statistics records (Fig. 4). In our example scenario, we observe 8 ABAP tasks, triggered by incoming HTTP server requests. One of them clearly dominates the load on the back end, and this is obviously due to database requests . The corresponding Top N Table Accesses sub record in the main statistics record indicates that sequential reads from two database tables account for almost all the time.
Figure 3: Transaction STATS_FE displays the front-end statistics records corresponding to user interactions. Fields with a green background characterize the front-end activity. Fields with a gray background contain measurements taken on the browser by the UI framework. The fields with the yellow background (normally shown to the right, here copied into an overlay) show aggregated values of the associated back-end statistics records.
The first record corresponds to the initial logon to the SAP S/4HANA Product Home Page. Then, 11 pairs of records represent calls from there to the Audit Journal and back. Observe that the first call to the application is significantly slower, presumably because buffers and caches have not been filled. Also note that only multiple repeated executions allow the identification of a failed measurement (crossed out in the screen shot).
The red arrows mark the front-end statistics record analyzed in more detail (Fig. 4).

 

Figure 4: A double-click on the front-end statistics record marked by the red arrows in Fig. 3 shows the associated back-end statistics records in transaction STATS. This resolves the back-end load triggered by the user interaction into separate tasks. Analyze the critical tasks, i.e., those that are slow or consume many resources. The records' details provide deeper insights and suggest the next steps in a root cause analysis. These details are displayed in a popup by double-click on the back-end statistics record.

 

For this scenario, our investigation so far has used solely the monitoring transactions STATS_FE and STATS based on the statistics records, which are always available. As a result, we know that one of the five calls to OData service /sap/opu/odata/sap/FAC_AUDIT_JOURNAL_SRV dominates the performance, and that accesses to tables IFIACCDOCJRNL and IFICMPTJRNL are the decisive factors (Fig. 4). To come closer to the root cause, we will use this information, together with knowledge of the back-end user's name, to set Conditions for Trace Recording in ST05 (Fig. 5).

Figure 5: Conditions for Trace Recording for a focussed investigation of Fiori application F0997 (Audit Journal), deduced from the monitoring data contained in front-end and back-end statistics records (Fig. 3 and Fig. 4).

 

The resulting trace (Fig. 6) is very much to the point. It contains only the HTTP requests serving the business logic. Those that were triggered by any generic framework did not pass the filter. With respect to SQL statements, exactly those that fetch records from the specified CDS views IFIACCDOCJRNL and IFICMPTJRNL were recorded. From here, we can focus on understanding how and where the statements are implemented, how they are called via various involved frameworks, and most importantly, how the database server has processed them. I cover all this in my blog post ST05: Analyze Individual Trace Records.

Figure 6: Complete trace of the start of the Audit Journal Fiori app recorded with the filter conditions shown in Fig. 5. Without filter, the trace for the same user interaction would contain several hundred SQL trace records.

 

Summary

The option to Activate Trace with Filter significantly enhances the basic trace recording capability of ST05 to capture the communications and data transfers between the ABAP work process and external components. Without it many performance analysis projects would be much more complicated, and others would even be impossible.
The particular choice of filter conditions shall be guided by previous measurements of the application under analysis, e.g., with transactions STATS or STATS_FE based on the statistics records. They identify the critical areas, and with corresponding filter criteria you limit the trace recording to these parts. This facilitates your subsequent investigation.
Trace Scheduling is really convenient for the analysis of applications that run at known after-hours times.