Jlaunch performance issue:
When issue reported over portal slowness ,during investigating it has been concluded that culprit process is jlaunch spiking to 100% CPU utilization.
How to troubleshoot:
1..Get the culprit <PID>
2..Execute ps -mp 32178208 -o THREAD
It will show the thread having CPU more than 100%
USER PID PPID TID ST CP PRI SC WCHAN F TT BND COMMAND
3.Now as per note 710154 create thread dump.
Use the JCMon tool to initiate a thread dump.
3(a). Start the JCMon program with the profile of the instance.
The JCMon binary is located in the /usr/sap/<SID>/<Instance>/j2ee/os_libs directory or in the /usr/sap/<SID>/SYS/exe/run directory.
3(b) Enter "20" to go to the "Local Administration Menue".
3(c) Choose the "Dump stacktrace" menu item by entering its number.
3(d) Select the Java node from the list of processes above and enter the value from the "Idx" column.
3(e) Confirm the command with a "y" if the selected node is correct.
After completion, you will find the thread dump:
in the /usr/sap/<SID>/<Instance>/work/std_<node name>.out file (for JDK's from SUN and HP, according to the patch level of jlaunch also for Classic VM on IBM i)
for JDK's from IBM: javacore<PID>.<timestamp>.txt in directory
/usr/sap/<SID>/<Instance>/j2ee/cluster/dispatcher
/usr/sap/<SID>/<Instance>/j2ee/cluster/server#
/usr/sap/<SID>/<Instance>/SDM/program
according to the patchlevel of jlaunch in a spoolfile QDMPJVM of the workprocess user (for Classic VM on IBM i)
4.. When you will check the thread dump in std_<Nodename>.out and dev_<Nodename> it will show following logs:
CMS: abort preclean due to time 6944417.054: [CMS-concurrent-abortable-preclean: 2.777/5.061 secs] [Times: user=0.00 sys=1.82, real=5.06 secs]
- 6944417.057: [GC[YG occupancy: 20912 K (512000 K)]6944417.057: [Rescan (parallel) , 0.0354850 secs]6944417.092: [weak refs processing, 0.0028530 secs]6944417.095: [class unloading, 0.0832610 secs]6944417.179: [scrub symbol table, 0.4329450 secs]6944417.612: [scrub string table, 0.0546810 secs] [1 CMS-remark: 795058K(3579904K)] 815970K(4091904K), 0.6875040 secs] [Times: user=0.00 sys=0.29, real=0.69 secs]
- 6944417.744: [CMS-concurrent-sweep-start]
- 6944418.331: [CMS-concurrent-sweep: 0.586/0.586 secs] [Times: user=0.00 sys=0.37, real=0.59 secs]
- 6944418.331: [CMS-concurrent-reset-start]
- 6944418.366: [CMS-concurrent-reset: 0.035/0.035 secs] [Times: user=0.00 sys=0.01, real=0.04 secs]
- 6944420.380: [GC [1 CMS-initial-mark: 795057K(3579904K)] 816280K(4091904K), 0.0410730 secs] [Times: user=0.00 sys=0.03, real=0.04 secs]
"Thread-547" daemon cpu=2.12 [reset 2.12] ms allocated=100424 B (98.07 kB) [reset 100424 B (98.07 kB)] defined_classes=0
io= file i/o: 0/0 B, net i/o: 30165/0 B, files opened:0, socks opened:0 [reset file i/o: 0/0 B, net i/o: 30165/0 B, files opened:0, socks opened:0 ]
prio=6 tid=0x0000000125afb800 nid=0x6b201a7 / 112329127 pthread-id=26161 runnable [_thread_in_native (_at_safepoint), stack(0x000000012bc1c888,0x000000012be1c888)] [0x000000012be1b000]
at java.net.SocketInputStream.socketRead0(Ljava/io/FileDescriptor;[BIII)I(Native Method)
at java.net.SocketInputStream.read([BII)I(SocketInputStream.java:129)
- additional info (remote: euws0036.biz.electrolux.com/10.5.32.59:389, local: localhost/127.0.0.1:50238)
at java.io.BufferedInputStream.fill()V(BufferedInputStream.java:183)
at java.io.BufferedInputStream.read1([BII)I(BufferedInputStream.java:222)
at java.io.BufferedInputStream.read([BII)I(BufferedInputStream.java:277)
- locked <0x000000023f4009e8> (a java.io.BufferedInputStream)
at com.sun.jndi.ldap.Connection.run()V(Connection.java:795)
at java.lang.Thread.run()V(Thread.java:676)
Locked ownable synchronizers:
- None
5.. So issue lies in long running Concurrent Mark-Sweep GC Thread which is basically caused by SWAP memory usage.
Do check following mark
A.Whether or not there's enough physical memory (RAM) on the server. (What's the size of the physical
memory?
B.How many instance/server nodes are installed, and what is the
heap size respectively?
6..And add parameter (“-XX:+UseCMSInitiatingOccupancyOnly instructs the CMS to start only
when the Java heap is filled up to 80%”) to server<X> node and restart the system.
This will resolve the issue.
Best Regards
Vaibhav
- SAP Managed Tags:
