Skip to Content

/wp-content/uploads/2014/04/tomcat77_424374.png

Troubleshooting CPU bottlenecks on the Apache Tomcat application server can be challenging.  Since Apache Tomcat is running as a Java Virtual Machine, understanding which JVM threads are consuming the overall Tomcat CPU can be a bit of a black box since this information is not easily accessible in regular monitoring tools.  Recently, I have discovered a brilliant new tool called JVMTOP which provides direct insight into the performance of threads running on a Java Virtual Machine.  Additionally, JVMTOP also shows in real-time which threads are blocked and also the id of the thread it is currently waiting on.  In this article, I will provide a demonstration on how this tool works and what information you need to collect once you have identified the problematic Java thread(s) so that SAP can perform a deeper root cause analysis.  Note that although this example is based on Apache Tomcat, JVMTOP will also work when troubleshooting CPU issues with the Adaptive Processing Server / Adaptive Job Server (child processes).

Identify the CPU bound thread using JVMTOP

      

I have made some modifications to the JVMTOP application (updated the executable jar manifest for ease of use) and created a new batch file to launch the application.  Download the version linked from this article when following this tutorial.  Note, if your Apache Tomcat currently uses Oracle JVM, you should configure it to use SAP JVM instead.  For instructions, refer to How to configure Tomcat to use SAP JVM instead of Oracle JVM – Business Intelligence (BusinessObjects) – SCN Wiki


  1. Download jvmtop.zip
     
  2. Extract the files from jvmtop.zip to <INSTALL_DIR>\SAP BusinessObjects Enterprise XI 4.0\win64_x64\sapjvm\bin on your Apache Tomcat node
     
  3. Open a command prompt and browse to <INSTALL_DIR>\SAP BusinessObjects Enterprise XI 4.0\win64_x64\sapjvm\bin
     
  4. Open task manager, click view –> select columns, select PID (Process Identifier) and determine the process id of tomcat.exe
     
    /wp-content/uploads/2014/04/tomcatt1_1_424311.png

  5. In the command prompt, execute the command: runJVMTOP.bat <PID>.  For example:

    /wp-content/uploads/2014/04/tomcatt3_424304.png

  6. JVMTOP will connect to the SAPJVM and list the top 10 Java threads which are consuming the most CPU.  Notice, it will continue to stay connected and the values are refreshed automatically in real-time similar to the Unix version of top/wp-content/uploads/2014/04/tomcatt5_424305.png

    COLUMN DESCRIPTION
    TID JVM thread ID
    NAME Name of JVM thread
    STATE The state that the thread is currently in
    CPU Current CPU utilization (in ratio to available cpu time on all processors)
    TOTALCPU CPU utilization (in ratio to process cpu consumption) since the thread is alive
    BLOCKEDBY If this thread is blocked, this column shows the thread id which is blocking this thread


  7. In the above example, we can easily identify that the thread creating the CPU bottleneck is http-8080-5 with thread id 48

Create thread dumps using JVMMON

Now that we have identified the offending thread, we need to generate a thread dump so that we can analyze the call stack of the code currently being executed by this thread.  SAPJVM includes a utility called JVMMON which makes creating thread dumps very easy to do without a lot of additional configuration.


  1. Open a command prompt and browse to <INSTALL_DIR>\SAP BusinessObjects Enterprise XI 4.0\win64_x64\sapjvm\bin
  2. Execute the file jvmmon.exe and it will print out a list of running JVM processes and their corresponding process id (PID)
  3. Enter the process id of the Apache Tomcat Server JVM and press enter

    /wp-content/uploads/2014/04/tomcat7_424312.png
  4. Enter the command “print to file” and when prompted for “file=” enter a path and file name for the first thread dump. NOTE: You must first create the folder structure where you intend to save the file.

  5. Next, enter the command “print stacktrace” and at the same time, take a screenshot of the current JVMTOP output. 

    /wp-content/uploads/2014/04/tomcat10_424340.png

    The threads shown in JVMTOP will match the thread details shown in the thread dump and then deeper root cause analysis can be performed by SAP

    /wp-content/uploads/2014/04/tomcat12_424341.png

  6. Repeat the thread dump / JVMTOP screenshot collection while the CPU is bottlenecked (at an interval of five minutes between each collection).  The more iterations of this information that is collected, the easier it will be to diagnose the bottleneck

  7. Exit JVMTOP by typing the keyboard combination CTRL-C and choose (Y) to end the batch job.  IMPORTANT: Do not click the X to close the command prompt window while JVMTOP is running.  Use the command exit to exit from JVMMON
       

      /wp-content/uploads/2014/04/tomcat13_424342.png


For more information about JVMTOP, you can visit the homepage at:

Documentation – jvmtop – Java monitoring for the command-line – Google Project Hosting



To report this post you need to login first.

19 Comments

You must be Logged on to comment or reply to a post.

  1. Arvind Pandalai

    This looks really cool…but can you share as mentioned by Denis the analysis and how can we remove the specific TID thread only which creates the bottleneck.

    (0) 
  2. Dave Cucura

    Thanks for the update clarifying the SAPJVM configuration! Maybe one day it could be possible to integrate the JVMTOP monitoring into the BI Platform Support Tool to create a wizard style interface for the process and standardize the capture timing and naming convention of the captured data’s filenames. This is an amazing tool for getting RCA on Tomcat CPU issues as-is though.

    (0) 
    1. Toby Johnston Post author

      Actually we are looking at integrating some of this functionality into the support tool.  This as well as an automatic thread dump collector.  It will probably be later this year or early 2015 as we currently have a full load of features lined up already

      (0) 
    1. Toby Johnston Post author

      Hi Raj,

      Technically yes however jvmmon comes only with SAPJVM.  You would need to use something like jstack instead to generate thread dumps from Oracle JVM.

      Regards

      Toby

      (0) 
  3. Stephen Folan

    Hi Toby,

    First off, great post. This has been invaluable to me over the past few months. Diagnosed a Tomcat issue in June. Thank you.

    I’ve been trying to get it to work with the Job Server on my internal system. I get some errors like ‘#’ and ‘#set’ are not recognised batch commands and followed by ‘JVMTOP cannot attache to the Job Server PID as it may have insufficient memory or privillages’.

    In the same system, JVMTOP works for Tomcat and APS no issues. Am I missing something when using with the Job Server? I don’t see the AJS PID in the JVM monitoring tool, leading me to believe that the Job Server doesn’t use a JVM.

    I’ve attached the cmd errors and the JVMMON-GUI screenshots.

    Thanks,

    StephenJVMMON_GUI.pngJVMTOP.JVMMON.SCN.png

    (0) 
    1. Toby Johnston Post author

      Hi Stephen,

      Your right, I had a slight error in the documentation.  The AJS itself is not a Java process however, when it processes a scheduled instance it spawns child processes which are Java (and can be connected to via JVMTOP)

      I have adjusted the documentation accordingly 🙂

      Thanks

      Toby

      (0) 
  4. Neeraj Bhadani

    Hi Toby,

           Thanks for the wonderful Blog. Do we have anything similar to Unix flavour. We are using AIX in our project and facing similar issue where one of the Java process is consuming most of the CPU.

    (0) 
    1. Toby Johnston Post author

      Hello Neeraj

      You can use JVMTOP on AIX too.  However, instead of using the batch file I provide to start JVMTOP, you will need to instead use the included shell script jvmtop.sh to start the program.  For example:


      jvmtop.sh 3539   


      (where 3539 is the process id of the jvm you want to connect to)

       JvmTop 0.4.1 alpha   amd64,  4 cpus, Linux 2.6.18-34
      http
      ://code.google.com/p/jvmtop

      PID
      3539: org.apache.catalina.startup.Bootstrap
      ARGS
      : start
      VMARGS
      : -Djava.util.logging.config.file=/home/webserver/apache-tomcat-5.5[...]
      VM
      : Sun Microsystems Inc. Java HotSpot(TM) 64-Bit Server VM 1.6.0_25
      UP
      : 869:33m #THR: 106  #THRPEAK: 143  #THRCREATED: 128020 USER: webserver
      CPU
      :  4.55% GC:  3.25% HEAP: 137m / 227m NONHEAP:  75m / 304m
      Note: Only top 10 threads (according cpu load) are shown!

        TID   NAME                                    STATE    CPU  TOTALCPU BLOCKEDBY
          
      25 http-8080-Processor13                RUNNABLE  4.55%     1.60%
      128022 RMI TCP Connection(18)-10.101.       RUNNABLE  1.82%     0.02%
       
      36578 http-8080-Processor164               RUNNABLE  0.91%     2.35%
       
      36453 http-8080-Processor94                RUNNABLE  0.91%     1.52%
          
      27 http-8080-Processor15                RUNNABLE  0.91%     1.81%
          
      14 http-8080-Processor2                 RUNNABLE  0.91%     3.17%
      128026 JMX server connection timeout   TIMED_WAITING  0.00%     0.00%
      128025 JMX server connection timeout   TIMED_WAITING  0.00%     0.00%
      128024 JMX server connection timeout   TIMED_WAITING  0.00%     0.00%
      128023 JMX server connection timeout   TIMED_WAITING  0.00%     0.00%

      Don’t forget to rate the article if it is helpful!

      Cheers

      Toby

      (0) 
  5. Mandanna M.U

    Hi Toby,

             This is a very good document for people like me , who are new to the world of Performance monitoring.

    While trying to follow the steps w.r.t. JVMTOP , i get an exception that the :”Exception class” is not found. As this is pretty new to me, request you to guide me in detail, on how to solve this class not found.

    Thanks for your time in advance.

    Regards,

    Mandanna

    Capture.PNG

    Capture.PNG

    I also tried downloading the setup from this link Downloads – jvmtop – Java monitoring for the command-line – Google Project Hosting

    But now i am getting some other error as seen below.

    Capture.PNG

    (0) 
    1. Toby Johnston Post author

      Try extracting the contents of jvmtop.zip directly into the bin folder instead of into the \bin\jvmtop subfolder as you have shown in your screenshot

      (0) 
  6. Ilyas Mohammed

    Hi Toby,

    Just wanted to know if this functionality has already been integrated with the BIPST tool version 2.0.2 ? Or is it still in the future plan?

    Appreciate your response.

    Thank you,

    Ilyas Mohammed

    (0) 

Leave a Reply