Technical Articles
Java Heap and Thread Dumps on SAP BTP Cloud Foundry
Java heap dumps and thread dumps are a powerful way to investigate what’s going on in the Java Virtual Machine (JVM). A heap dump allows you to see what is kept in memory after a minor and major garbage collection. Therefore, heap dumps provide indispensable insights which can be used for troubleshooting memory leaks.
This blog post is the third and final part (following part 1 and part 2) of our blog series focusing on Java memory management in container environments. We will show how to create heap dumps and thread dumps on demand for Java applications running on SAP BTP Cloud Foundry.
The cf java plugin
Heap dumps can be created on demand by requesting them from the JVM via tools like jvmmon
or jcmd
. Which tool and command to use depends on the JVM vendor as well as the Java buildpack which is used on SAP BTP Cloud Foundry. To make it as simple as possible to create heap dumps and thread dumps on demand, we’ve created the cf java
plugin plugin.
The cf java
plugin is installed as an addition to the Cloud Foundry Command Line Interface (CF CLI), which is used to manage applications on SAP BTP Cloud Foundry.
You can add the cf java
plugin via the command line:
$ cf add-plugin-repo CF-Community http://plugins.cloudfoundry.org
$ cf install-plugin -r CF-Community "java"
Once installed, you can call the cf java
plugin. To view all commands and options supported by the plugin, you can run cf java --help
:
$ cf java --help NAME: java - Obtain a heap-dump or thread-dump from a running, SSH-enabled Java application. USAGE: cf java [heap-dump|thread-dump] APP_NAME OPTIONS: --app-instance-index -i [index], select to which instance of the app to connect --container-dir -cd, the directory path in the container that the heap dump file will be saved to --dry-run -n, just output to command line what would be executed --keep -k, keep the heap dump in the container; by default the heap dump will be deleted from the container's filesystem after been downloaded --local-dir -ld, the local directory path that the dump file will be saved to
The cf java
plugin uses an SSH tunnel to connect to a selected application APP_NAME
and instance --app-instance-index
, to trigger a heap dump or thread dump with the JVM specific tools and commands. That said, you need to ensure that your application has the necessary JDK tools installed and SSH access to your application is enabled. By default, SSH access is disabled:
$ cf ssh-enabled APP_NAME ssh is disabled for app
You can enable SSH access to your application APP_NAME
via the following command and a restart of your application:
$ cf enable-ssh APP_NAME $ cf restart APP_NAME
Another important consideration is the disk space which is available to your application. By default, every application gets a disk of 1 GB attached to the application container. In theory, a heap dump of a Java app can get as large as the maximum heap size (-Xmx
) configured for the JVM. Therefore, it’s not unusual for a heap dump to exceed 1 GB of the available disk space. You can request more disk space via the following property in your manifest file (with a maximum total disk space of 4 GB):
--- ... disk_quota: 4G
Note that a change to the manifest file requires you to push this change via cf push
. If you don’t have the manifest file at hand, you can also use the following command to increase the disk quota:
$ cf scale APP_NAME -k 4G
After checking off all the prerequisites, you can create a heap dump using the cf java
plugin and transfer it to your local computer:
$ cf java heap-dump APP_NAME --local-dir FOLDER
For example, to create a heap dump of the application with the name spring-music
and save it into the current folder on your local computer, you can run:
$ cf java heap-dump spring-music --local-dir . Successfully created heap dump in application container at: /var/vcap/data/fcb80119-7117-4682-8285-d7d4740cdf66/java_pid12_5.hprof Heap dump filed saved to: ./spring-music-heapdump-231be9d8-d3a5-403f-aa44-14637c4510d3.hprof Heap dump filed deleted in app container
With the generated heap dump, you can now use a heap analyser tool of your choice, for example the Eclipse Memory Analyzer (MAT).
It is also possible to create a thread dump by using the thread-dump
command:
$ cf java thread-dump APP_NAME > thread.dump
With this setup, you are now able to create heap dumps and thread dumps in no time!
Great blog Tim.
Can you suggest the right tool for live monitoring especially to analyze heap, and gc logs.
Hi Mohammad Saad Rashid, thank you for your kind feedback.
There are several ways to get live monitoring of JVM metrics:
cf ssh
after enabling JMX extension through the buildpack configuration (let me know if you want more details as the configuration is a little bit tricky).Let me know if you have any further questions.
Hi Tim Gerlach ,
Blog is really good.Can you please provide more details on best Heap analyser tool and live monitoring of JVM metrics.
Thank you.
Hi Gaurish Shah,
Thanks for your feedback.
If you already have heap dump you can use the Memory Analysis Tool from Eclipse as mentioned at the end of the blog.
Starting with Java 11 you Application Performance Monitoring tools like Dynatrace can also provide you with a continuous (live) heap analysis which make the need for a heap dump a little less important.
Best regards,
Tim
Hi Tim,
This blog was very helpful in understanding ssh to cf app. However after enable-ssh and restart, I am still getting "ssh is not enabled for app". I have tried restart, restage, and even pushing the app again. But no luck. What else could be wrong?
Thanks,
Ram
After setting the java version in the manifest file and pushing the app again, I am able to collect the dumps.
ex:
Thanks,
Ram
Hi Rama,
thanks a lot for your valuable feedback. I have updated the blog and added a link to the JDK tool requirements to make sure that all prerequisites are mentioned in this blog.
Thanks,
Tim
Hi Tim.
for my application, in CF, it shows, 2GB is consumed by the application.
when I create a heap dump for it . it shows only 450MB used.
do you have any idea how can I get the real memory allocated?
Regards
Eric
Hi Eric,
The "Memory" shows the OS level memory usage, it is allocated/reserved by the JVM.
When a heap dump is generated, the JVM performs a full GC, then it saves the heap content to the dump file.
SAP Note 1824799 - Memory usage of the SAP JVM
provides some details.
Best regards,
Antal
Hi Tim,
Thank you for the blog... All the steps are neatly explained..
I tried to take heap dump of our application and incresed the disk space to 4G.
After that also I am getting below error:
Dumping heap to /var/vcap/data/xyz/app-nameheapdump-c14e88da-f2f3-4cbd-b71f-d896e5fb3a40.hprof ... Unable to create /var/vcap/data/xyz/app-name-heapdump-c14e88da-f2f3-4cbd-b71f-d896e5fb3a40.hprof: No space left on device
Kindly check and help us..
is increasing memory is a option??
Regards,
Lekhak Patil
Hi Lekhak,
depending on the size of the JVM there should be enough on the disk for at least one heap dump. Can you please check that the app is entitled to use 4G of disk size and verify via ssh if there's already any other heap dump that was created on the disk. You can check via
cf ssh APP_NAME
. If this doesn't help, can you pleas e let me know how much memory is assigned to your application?Best regards,
Tim