Toby Johnston

April 2, 2014 4 minute read

Using JVMTOP and JVMMON to analyze Tomcat and Adaptive Processing Server CPU bottlenecks

Troubleshooting CPU bottlenecks on the Apache Tomcat application server can be challenging. Since Apache Tomcat is running as a Java Virtual Machine, understanding which JVM threads are consuming the overall Tomcat CPU can be a bit of a black box since this information is not easily accessible in regular monitoring tools. Recently, I have discovered a brilliant new tool called JVMTOP which provides direct insight into the performance of threads running on a Java Virtual Machine. Additionally, JVMTOP also shows in real-time which threads are blocked and also the id of the thread it is currently waiting on. In this article, I will provide a demonstration on how this tool works and what information you need to collect once you have identified the problematic Java thread(s) so that SAP can perform a deeper root cause analysis. Note that although this example is based on Apache Tomcat, JVMTOP will also work when troubleshooting CPU issues with the Adaptive Processing Server / Adaptive Job Server (child processes).

Identify the CPU bound thread using JVMTOP

I have made some modifications to the JVMTOP application (updated the executable jar manifest for ease of use) and created a new batch file to launch the application. Download the version linked from this article when following this tutorial. Note, if your Apache Tomcat currently uses Oracle JVM, you should configure it to use SAP JVM instead. For instructions, refer to How to configure Tomcat to use SAP JVM instead of Oracle JVM – Business Intelligence (BusinessObjects) – SCN Wiki

Download jvmtop.zip
Extract the files from jvmtop.zip to <INSTALL_DIR>\SAP BusinessObjects Enterprise XI 4.0\win64_x64\sapjvm\bin on your Apache Tomcat node
Open a command prompt and browse to <INSTALL_DIR>\SAP BusinessObjects Enterprise XI 4.0\win64_x64\sapjvm\bin
Open task manager, click view –> select columns, select PID (Process Identifier) and determine the process id of tomcat.exe
In the command prompt, execute the command: runJVMTOP.bat <PID>. For example:

JVMTOP will connect to the SAPJVM and list the top 10 Java threads which are consuming the most CPU. Notice, it will continue to stay connected and the values are refreshed automatically in real-time similar to the Unix version of top

/wp-content/uploads/2014/04/tomcatt5_424305.png

COLUMN	DESCRIPTION
TID	JVM thread ID
NAME	Name of JVM thread
STATE	The state that the thread is currently in
CPU	Current CPU utilization (in ratio to available cpu time on all processors)
TOTALCPU	CPU utilization (in ratio to process cpu consumption) since the thread is alive
BLOCKEDBY	If this thread is blocked, this column shows the thread id which is blocking this thread

In the above example, we can easily identify that the thread creating the CPU bottleneck is http-8080-5 with thread id 48

Create thread dumps using JVMMON

Now that we have identified the offending thread, we need to generate a thread dump so that we can analyze the call stack of the code currently being executed by this thread. SAPJVM includes a utility called JVMMON which makes creating thread dumps very easy to do without a lot of additional configuration.

Open a command prompt and browse to <INSTALL_DIR>\SAP BusinessObjects Enterprise XI 4.0\win64_x64\sapjvm\bin
Execute the file jvmmon.exe and it will print out a list of running JVM processes and their corresponding process id (PID)
Enter the process id of the Apache Tomcat Server JVM and press enter
Enter the command “print to file” and when prompted for “file=” enter a path and file name for the first thread dump. NOTE: You must first create the folder structure where you intend to save the file.
Next, enter the command “print stacktrace” and at the same time, take a screenshot of the current JVMTOP output.

The threads shown in JVMTOP will match the thread details shown in the thread dump and then deeper root cause analysis can be performed by SAP
Repeat the thread dump / JVMTOP screenshot collection while the CPU is bottlenecked (at an interval of five minutes between each collection). The more iterations of this information that is collected, the easier it will be to diagnose the bottleneck
Exit JVMTOP by typing the keyboard combination CTRL-C and choose (Y) to end the batch job. IMPORTANT: Do not click the X to close the command prompt window while JVMTOP is running. Use the command exit to exit from JVMMON

For more information about JVMTOP, you can visit the homepage at:

Documentation – jvmtop – Java monitoring for the command-line – Google Project Hosting

29 Comments

You must be Logged on to comment or reply to a post.

Brian Thomas

April 2, 2014 at 8:29 pm

Awesome post. I always wanted to find a tool that would do this.

Denis Konovalov

April 3, 2014 at 4:05 pm

Awesome. How about analyzing what has just been collected ?

Matthew Shaw

May 8, 2014 at 6:59 pm

Another great blog Toby

Former Member

June 12, 2014 at 9:24 am

This looks really cool...but can you share as mentioned by Denis the analysis and how can we remove the specific TID thread only which creates the bottleneck.

Dave Cucura

August 7, 2014 at 4:00 pm

Thanks for the update clarifying the SAPJVM configuration! Maybe one day it could be possible to integrate the JVMTOP monitoring into the BI Platform Support Tool to create a wizard style interface for the process and standardize the capture timing and naming convention of the captured data's filenames. This is an amazing tool for getting RCA on Tomcat CPU issues as-is though.

Toby Johnston

Blog Post Author

August 7, 2014 at 7:35 pm

Actually we are looking at integrating some of this functionality into the support tool. This as well as an automatic thread dump collector. It will probably be later this year or early 2015 as we currently have a full load of features lined up already

Former Member

August 15, 2014 at 2:56 pm

Thanks for the tool.. Question, does it work for BO 3.1 tomcat ?

Toby Johnston

Blog Post Author

August 18, 2014 at 7:27 pm

Hi Raj,

Technically yes however jvmmon comes only with SAPJVM. You would need to use something like jstack instead to generate thread dumps from Oracle JVM.

Regards

Toby

Stephen Folan

August 29, 2014 at 9:25 am

Hi Toby,

First off, great post. This has been invaluable to me over the past few months. Diagnosed a Tomcat issue in June. Thank you.

I've been trying to get it to work with the Job Server on my internal system. I get some errors like '#' and '#set' are not recognised batch commands and followed by 'JVMTOP cannot attache to the Job Server PID as it may have insufficient memory or privillages'.

In the same system, JVMTOP works for Tomcat and APS no issues. Am I missing something when using with the Job Server? I don't see the AJS PID in the JVM monitoring tool, leading me to believe that the Job Server doesn't use a JVM.

I've attached the cmd errors and the JVMMON-GUI screenshots.

Thanks,

Stephen

Toby Johnston

Blog Post Author

August 29, 2014 at 2:46 pm

Hi Stephen,

Your right, I had a slight error in the documentation. The AJS itself is not a Java process however, when it processes a scheduled instance it spawns child processes which are Java (and can be connected to via JVMTOP)

I have adjusted the documentation accordingly 🙂

Thanks

Toby

Giridhara Tadikonda

March 14, 2016 at 9:11 pm

I am getting error with access denied when I tried. do you think I missed providing access after I copied the files?

Fetih Köroglu

October 22, 2019 at 10:09 am

Hello Mr. Tadikonda,

Could you find a solution to this problem? I have same Access denied error.

regards,

Fetih

Former Member

January 7, 2015 at 3:26 pm

Hi Toby,

Thanks for the wonderful Blog. Do we have anything similar to Unix flavour. We are using AIX in our project and facing similar issue where one of the Java process is consuming most of the CPU.

Toby Johnston

Blog Post Author

January 7, 2015 at 4:15 pm

Hello Neeraj

You can use JVMTOP on AIX too. However, instead of using the batch file I provide to start JVMTOP, you will need to instead use the included shell script jvmtop.sh to start the program. For example:

jvmtop.sh 3539

(where 3539 is the process id of the jvm you want to connect to)

 JvmTop 0.4.1 alpha   amd64,  4 cpus, Linux 2.6.18-34
 http://code.google.com/p/jvmtop

 PID 3539: org.apache.catalina.startup.Bootstrap
 ARGS: start
 VMARGS: -Djava.util.logging.config.file=/home/webserver/apache-tomcat-5.5[...]
 VM: Sun Microsystems Inc. Java HotSpot(TM) 64-Bit Server VM 1.6.0_25
 UP: 869:33m #THR: 106  #THRPEAK: 143  #THRCREATED: 128020 USER: webserver
 CPU:  4.55% GC:  3.25% HEAP: 137m / 227m NONHEAP:  75m / 304m
 Note: Only top 10 threads (according cpu load) are shown!

  TID   NAME                                    STATE    CPU  TOTALCPU BLOCKEDBY
     25 http-8080-Processor13                RUNNABLE  4.55%     1.60%
 128022 RMI TCP Connection(18)-10.101.       RUNNABLE  1.82%     0.02%
  36578 http-8080-Processor164               RUNNABLE  0.91%     2.35%
  36453 http-8080-Processor94                RUNNABLE  0.91%     1.52%
     27 http-8080-Processor15                RUNNABLE  0.91%     1.81%
     14 http-8080-Processor2                 RUNNABLE  0.91%     3.17%
 128026 JMX server connection timeout   TIMED_WAITING  0.00%     0.00%
 128025 JMX server connection timeout   TIMED_WAITING  0.00%     0.00%
 128024 JMX server connection timeout   TIMED_WAITING  0.00%     0.00%
 128023 JMX server connection timeout   TIMED_WAITING  0.00%     0.00%

Don't forget to rate the article if it is helpful!

Cheers

Toby

Mandanna Mandechanda Uthappa

June 11, 2015 at 10:53 am

Hi Toby,

This is a very good document for people like me , who are new to the world of Performance monitoring.

While trying to follow the steps w.r.t. JVMTOP , i get an exception that the :"Exception class" is not found. As this is pretty new to me, request you to guide me in detail, on how to solve this class not found.

Thanks for your time in advance.

Regards,

Mandanna

I also tried downloading the setup from this link Downloads - jvmtop - Java monitoring for the command-line - Google Project Hosting

But now i am getting some other error as seen below.

Toby Johnston

Blog Post Author

June 12, 2015 at 5:45 pm

Try extracting the contents of jvmtop.zip directly into the bin folder instead of into the \bin\jvmtop subfolder as you have shown in your screenshot

Mandanna Mandechanda Uthappa

June 15, 2015 at 4:25 am

Thanks Toby,

It worked now...

Regards,

Mandanna

Former Member

March 14, 2016 at 2:50 pm

Hi Toby,

Just wanted to know if this functionality has already been integrated with the BIPST tool version 2.0.2 ? Or is it still in the future plan?

Appreciate your response.

Thank you,

Ilyas Mohammed

Toby Johnston

Blog Post Author

March 14, 2016 at 5:45 pm

Hi Ilyas,

Not yet, but it is something we are considering potentially for version 2.1

Cheers

Toby

Former Member

March 14, 2016 at 7:07 pm

That would be awesome and would potentially reduce the admin time for all administrators.

Thank you,

Ilyas

Praveen Chukka

October 10, 2019 at 3:05 pm

Hi Toby,

Just wanted to know if this functionality has already been integrated with the BIPST tool version 2.1 ?

Toby Johnston

Blog Post Author

October 10, 2019 at 6:45 pm

Hi Praveen,

No, we havent implemented this yet but definitely need to revisit this idea. I think we could make it much more useful (such as combining the jvmtop view from multiple tomcat nodes at once).

Will bring this up at our next sprint meeting.

Cheers

Toby

Praveen Chukka

October 11, 2019 at 5:57 pm

Thanks Toby. Yes, it will very nice to have it in support tool so we don't have to login to multiple tomcat nodes and collect the info. Please let us know the updates after your meeting. Thanks!

Praveen

Praveen Chukka

February 26, 2020 at 8:10 pm

Hi Toby - any updates on when this is going to integrate into BIP support tool? Thanks!

Praveen Chukka

October 14, 2019 at 6:34 pm

Hi Toby,

Can you also provide more information on analyzing what is collected using JVMMON? Thanks in advance!

Reinis Berzins

April 8, 2020 at 11:01 pm

Thanks for the great post. We are experiencing intermittent Tomcat 7 CPU spikes above 90% and I'm trying to use JVMTOP to get the problem thread.

JVMTOP works great on the server when I test it during a relatively calm period, but when the CPU issue is occurring the process appears to stall indefiinitely - after I execute RunJVMTOP.bat <pid> nothing happens in the console until I eventually Ctrl+C to terminate the job. I've left it running for 15 mins with no luck.

I am able to get the stack trace using the JVMMON but unfortunately that doesn't point me to the culprit.

I'd appreciate any suggestions - It's a bit frustrating that it works great until the one time I actually need to get information from it. 🙂

Thanks,

Reinis

Toby Johnston

Blog Post Author

May 11, 2020 at 2:45 pm

Hi Reinis,

It could be that Tomcat is completely consumed with stuck threads and as a result JVMTOP is no longer able to function. You might have better luck if you catch the problem earlier on before Tomcat goes completely unresponsive.

Either way, you should probably open a case with SAP Support to help on this issue. A series of thread dumps (spaced a couple minutes apart) would be helpful in troubleshooting the issue.

Thanks

Toby

SystemAdmin systemAdmin

May 11, 2020 at 2:39 pm

Hi Toby,

I've tried running this for the first time today and I have successfully created the file but I may have missed the fact that the CMC and BI Launchpad will become unavailable while you are generating the file, in my case it did, sadly.

I had to 'exit' the command prompt and had to stop and restart Tomcat to allow access to CMC and BI Launchpad again. Surely I did something wrong?

Your comments will be appreciated.

Thanks & regards

Cornelius

Toby Johnston

Blog Post Author

May 11, 2020 at 2:49 pm

Hi Cornelius,

When did CMC and Launchpad become unavailable exactly? Was it when running JVMMON or JVMTOP? If running JVMTOP and you don't gracefully exit the program it will cause Tomcat to shutdown. In this instance I could see it causing this problem. However, under normal circumstances, running JVMTOP will not prevent access to the apps running on the Tomcat server. Could it be that at that point the Tomcat was just unresponsive due to the issue you were troubleshooting?

Thanks
Toby

Using JVMTOP and JVMMON to analyze Tomcat and Adaptive Processing Server CPU bottlenecks

Identify the CPU bound thread using JVMTOP

Create thread dumps using JVMMON

Assigned Tags