GC analysis and troubleshooting with SapMachine
There are several possibilities to analyze and troubleshoot GC issues. You can activate the gc log via Unified JVM logging(-Xlog:gc) to get a textual log output. This can be also set for detailed GC phases and to specific levels. The analysis can be done on the textual log file or with available open source tools like JIFA or commercial tools like GCeasy to parse and analyze the log and visualize the findings.
We, from SapMachine development team, build and provide also a JMC version on SapMachine.
SapMachine will provide two additional Flight Recording configurations
The SapMachine starting with 11.0.19 and 17.0.7 provides two additional Flight Recording configurations(located in the directory lib/jfr of the JDK/JRE), especially for GC profiling.
The gc.jfc, which is a lightweight GC profiling. This can be used also for longer profiling runs and will provide general GC profiling data with limited details and small recording size.
The gc_details.jfc will record all GC events and details. This will have a higher impact caused by heap inspection initiated GCs to get e.g. heap statistics and have a large recording size.
How to do the JFR profiling:
In general there are three ways to do the recording:
- The Flight Recording can be enabled by JVM parameter to start this with the startup already(-XX:StartFlightRecording). Per option you can specify the filename to be used to save the recording and you can specify the settings/configuration file.
- Start the Flight Recorder via jcmd. Connect to the running Java application(jcmd <main class|PID>) and start/dump/stop the recording by specific command(JFR.start, JFR.dump, JFR.stop). You can specify the recording name, settings(configuration), etc. Check the help of the commands to get all available options.
In SAP BTP you have to enable ssh in CF first to call the jcmd via cf ssh.
cf ssh <your app name> -c "app/META-INF/.sap_java_buildpack/sap_machine_jre/bin/jcmd $(pgrep java) JFR.start name=gc_recording settings=gc_details.jfc filename=/home/vcap/tmp/gc.jfr"
- Connect with JMC to a running local or remote Java application to start/dump/stop the recording without restart. You have to open JMX protocol listener. This can be done with e.g. jcmd <main class|PID> ManagementAgent.start jmxremote.authenticate=false jmxremote.ssl=false jmxremote.port=5555.
If you want to profile a Java application running behind a firewall you may need port forwarding.
In SAP BTP you have to start your app with -Djava.rmi.server.hostname=127.0.0.1 and start the Management Agent via cf ssh and use a ssh tunnel. E.g.:
cf ssh <your app name> -c "app/META-INF/.sap_java_buildpack/sap_machine_jre/bin/jcmd $(pgrep java) ManagementAgent.start jmxremote.authenticate=false jmxremote.ssl=false jmxremote.port=5555 jmxremote.rmi.port=5555"
cf ssh <your app name> -N -T -L 5555:127.0.0.1:5555
Connect with JMC to 127.0.0.1:5555 to get the JMX Console or starting the Flight Recording.
Depending on what your SapMachine is using (JRE or JDK), specify the path accordingly (sap_machine_jre or sap_machine_jdk). To learn more, see: SapMachine.
How to analyze the JFR recording:
In addition JMC is providing specific Outlines e.g. Java Application, JVM Internals, Environment and the Event Browser. The Outlines will show the specific data in tables or graphs.
With the additional JFR configurations the SapMachine provides, general GC analysis with low overhead and also details GC analysis is possible out of the box as the configurations are delivered with the SapMachine.
In case of questions about the usage of JFR and JMC for GC analysis or other profiling don’t hesitate to contact me and any feedback about this post is very welcome.
Ask questions about SAP Java Virtual Machine and follow (https://answers.sap.com/tags/01200615320800003576)
Read other SAP Java Virtual Machine blog posts and follow (https://blogs.sap.com/tags/01200615320800003576/)