GC analysis and troubleshooting with SapMachine

[ansteiner]

There are several possibilities to analyze and troubleshoot GC issues. You can activate the gc log via Unified JVM logging(-Xlog:gc) to get a textual log output. This can be also set for detailed GC phases and to specific levels. The analysis can be done on the textual log file or with available open source tools like JIFA or commercial tools like GCeasy to parse and analyze the log and visualize the findings.

Another possibility is to use the Java Flight Recorder (JFR) to profile your Java application and analyze this with Java Mission Control (JMC).

We, from SapMachine development team, build and provide also a JMC version on SapMachine.

SapMachine will provide two additional Flight Recording configurations

The SapMachine starting with 11.0.19 and 17.0.7 provides two additional Flight Recording configurations(located in the directory lib/jfr of the JDK/JRE), especially for GC profiling.
The gc.jfc, which is a lightweight GC profiling. This can be used also for longer profiling runs and will provide general GC profiling data with limited details and small recording size.
The gc_details.jfc will record all GC events and details. This will have a higher impact caused by heap inspection initiated GCs to get e.g. heap statistics and have a large recording size.

How to do the JFR profiling:

In general there are three ways to do the recording:

1. The Flight Recording can be enabled by JVM parameter to start this with the startup already(-XX:StartFlightRecording). Per option you can specify the filename to be used to save the recording and you can specify the settings/configuration file.
   E.g. -XX:StartFlightRecording,filename=./my_recording.jfr,settings=gc_details.jfc


2. Start the Flight Recorder via jcmd. Connect to the running Java application(jcmd <main class|PID>) and start/dump/stop the recording by specific command(JFR.start, JFR.dump, JFR.stop). You can specify the recording name, settings(configuration), etc. Check the help of the commands to get all available options.
   In SAP BTP you have to enable ssh in CF first to call the jcmd via cf ssh.
   E.g.:
   

   cf ssh <your app name> -c "app/META-INF/.sap_java_buildpack/sap_machine_jre/bin/jcmd $(pgrep java) JFR.start name=gc_recording settings=gc_details.jfc filename=/home/vcap/tmp/gc.jfr"

   
   


3. Connect with JMC to a running local or remote Java application to start/dump/stop the recording without restart. You have to open JMX protocol listener. This can be done with e.g. jcmd <main class|PID> ManagementAgent.start jmxremote.authenticate=false jmxremote.ssl=false jmxremote.port=5555.
   If you want to profile a Java application running behind a firewall you may need port forwarding.
   In SAP BTP you have to start your app with -Djava.rmi.server.hostname=127.0.0.1 and start the Management Agent via cf ssh and use a ssh tunnel. E.g.:
   

   cf ssh <your app name> -c "app/META-INF/.sap_java_buildpack/sap_machine_jre/bin/jcmd $(pgrep java) ManagementAgent.start jmxremote.authenticate=false jmxremote.ssl=false jmxremote.port=5555 jmxremote.rmi.port=5555"

   
   
   
   

   cf ssh <your app name> -N -T -L 5555:127.0.0.1:5555

   
   Connect with JMC to 127.0.0.1:5555 to get the JMX Console or starting the Flight Recording.
   
   Note:
   
   Depending on what your SapMachine is using (JRE or JDK), specify the path accordingly (sap_machine_jre or sap_machine_jdk). To learn more, see: SapMachine.
   
    
   
   

How to analyze the JFR recording:

The collected jfr recording can be analyzed with JMC. Open the JFR recording file to get the Automated Analysis Results as a first overview.

Figure 1: Analysis overview

 

The provided findings can be expanded to get more details with hints for optimization.

Figure 2: Expanded analysis details with hints for optimization

 

In addition JMC is providing specific Outlines e.g. Java Application, JVM Internals, Environment and the Event Browser. The Outlines will show the specific data in tables or graphs.

In JVM Internals you will find the GC Summary with average, maximum, total GC times of the collections:

Figure 3: GC summary

 

The GC configuration details:

Figure 4: The GC configuration details

The Garbage Collections with the GC details:

Figure 5: Garbage collection details

Depending of the enabled events in the used JFR configuration, the table will show some or all the GC Phase Pause Levels with detailed pause time.

 

All collected events can be found in the Event Browser:

Figure 6: Event Browser

There you can also create your own specific outlines/pages for events your are interested in and no outline exists already. In addition you can create outlines/pages for your own custom events.

 

Conclusion

With the additional JFR configurations the SapMachine provides, general GC analysis with low overhead and also details GC analysis is possible out of the box as the configurations are delivered with the SapMachine.

In case of questions about the usage of JFR and JMC for GC analysis or other profiling don't hesitate to contact me and any feedback about this post is very welcome.

 

References:
Ask questions about SAP Java Virtual Machine and follow  (https://answers.sap.com/tags/01200615320800003576)

Read other SAP Java Virtual Machine blog posts and follow (https://blogs.sap.com/tags/01200615320800003576/)