A simple way to analyze thread contention problems in Java

One of the features that makes programming a complex web application difficult is, that you have to deal with multiple threads running concurrently. Writing correct multithreaded programming is not easy. Writing correct and efficient concurrent java programs is known to be even more difficult. A great book about concurrent programming has been written by Doug Lea. Doug Lea’s work has been incorporated into the new concurrent classes in JDK 1.5. A backport to JDK 1.4 is also available. If you have not yet taken a closer look at the new concurrent classes, I urge you to do so, because these classes simplify multithreaded programming in Java very much.

Analyzing performance problems on a production system I showed you how thread dumps can be used to get simple, not very accurate, but still useful information about bottlenecks in your applications.

This time I will show you a few commands than can help you to find bottlenecks that are caused by thread contention problems.
After you’ve got Analyzing performance problems on a production system (the more the better), you can use the following command

grep "waiting to lock" std_server0.out | sort -k2 -r | uniq -c |sort -k1 -n -r | more

to find the threads that are waiting for a lock and

grep " locked" std_server0.out | sort -k3 | uniq -c | sort -k 1 -n -r | more

to find the threads that are holding a lock.

You are interested in locks that a lot of Threads are waiting for, Suppose you get something like this from the first command :

15 - waiting to lock <0x1f785860> (a com.sap.StringConverter) 12 - waiting to lock <0x1f772f28> (a com.sap.StringConverter)

The lines above tell you that most likely there’s a bottleneck in com.sap.ShortStringConverter, because there are 27 (15+12) places where a thread is waiting because it can’t aquire a lock.

If you then run the second command, and you find something like this :

2 - locked <0x1f785860> (a com.sap.StringConverter) 1 - locked <0x1f774380> (a com.sap.StringConverter)

then you can be pretty sure that you’ve got a bottleneck. By the way I didn’t really made this example up. It’s taken from a real world example. I’ve only changed the name of the class, to protect my colleagues from being flamed 😉

The The SAP Java VM – What’s in it for me? will come soon with a more detailed thread dump format, that can help you to find threading problems more easily. I plan to blog on this as soon as the new version has been released.