Java Heap and Thread Dumps on SAP BTP Cloud Foundry

Java heap dumps and thread dumps are a powerful way to investigate what’s going on in the Java Virtual Machine (JVM). A heap dump allows you to see what is kept in memory after a minor and major garbage collection. Therefore, heap dumps provide indispensable insights which can be used for troubleshooting memory leaks.

This blog post is the third and final part (following part 1 and part 2) of our blog series focusing on Java memory management in container environments. We will show how to create heap dumps and thread dumps on demand for Java applications running on SAP BTP Cloud Foundry.

The cf java plugin

Heap dumps can be created on demand by requesting them from the JVM via tools like jvmmon or jcmd. Which tool and command to use depends on the JVM vendor as well as the Java buildpack which is used on SAP BTP Cloud Foundry. To make it as simple as possible to create heap dumps and thread dumps on demand, we’ve created the cf java plugin plugin.

The cf java plugin is installed as an addition to the Cloud Foundry Command Line Interface (CF CLI), which is used to manage applications on SAP BTP Cloud Foundry.

You can add the cf java plugin via the command line:

$ cf add-plugin-repo CF-Community http://plugins.cloudfoundry.org
$ cf install-plugin -r CF-Community "java"

Once installed, you can call the cf java plugin. To view all commands and options supported by the plugin, you can run cf java --help:

$ cf java --help 
NAME:
   java - Obtain a heap-dump or thread-dump from a running, SSH-enabled Java application.

USAGE:
   cf java [heap-dump|thread-dump] APP_NAME

OPTIONS:
   --app-instance-index      -i [index], select to which instance of the app to connect
   --container-dir           -cd, the directory path in the container that the heap dump file will be saved to
   --dry-run                 -n, just output to command line what would be executed
   --keep                    -k, keep the heap dump in the container; by default the heap dump will be deleted from the container's filesystem after been downloaded
   --local-dir               -ld, the local directory path that the dump file will be saved to

The cf java plugin uses an SSH tunnel to connect to a selected application APP_NAME and instance --app-instance-index, to trigger a heap dump or thread dump with the JVM specific tools and commands. That said, you need to ensure that your application has the necessary JDK tools installed and SSH access to your application is enabled. By default, SSH access is disabled:

$ cf ssh-enabled APP_NAME
ssh is disabled for app

You can enable SSH access to your application APP_NAME via the following command and a restart of your application:

$ cf enable-ssh APP_NAME
$ cf restart APP_NAME

Another important consideration is the disk space which is available to your application. By default, every application gets a disk of 1 GB attached to the application container. In theory, a heap dump of a Java app can get as large as the maximum heap size (-Xmx) configured for the JVM. Therefore, it’s not unusual for a heap dump to exceed 1 GB of the available disk space. You can request more disk space via the following property in your manifest file (with a maximum total disk space of 4 GB):

---
  ...
  disk_quota: 4G

Note that a change to the manifest file requires you to push this change via cf push. If you don’t have the manifest file at hand, you can also use the following command to increase the disk quota:

$ cf scale APP_NAME -k 4G

After checking off all the prerequisites, you can create a heap dump using the cf java plugin and transfer it to your local computer:

$ cf java heap-dump APP_NAME --local-dir FOLDER

For example, to create a heap dump of the application with the name spring-music and save it into the current folder on your local computer, you can run:

$ cf java heap-dump spring-music --local-dir .
Successfully created heap dump in application container at: /var/vcap/data/fcb80119-7117-4682-8285-d7d4740cdf66/java_pid12_5.hprof
Heap dump filed saved to: ./spring-music-heapdump-231be9d8-d3a5-403f-aa44-14637c4510d3.hprof
Heap dump filed deleted in app container

With the generated heap dump, you can now use a heap analyser tool of your choice, for example the Eclipse Memory Analyzer (MAT).

It is also possible to create a thread dump by using the thread-dump command:

$ cf java thread-dump APP_NAME > thread.dump

With this setup, you are now able to create heap dumps and thread dumps in no time!