After two chapters (you can read them The wonderful world of java threads (I) and The wonderful world of java threads (II)), it is time to close this blog series about the wonderful world of thread dumps. I will finish with two examples on the different situations we can find, but obviously there are quite a lot more. What is important here is to understand the concept of thread dump, to know that there are useful tools to check them and to see what is going on in our system when the problem arises.
Example 3: Webdynpro is making the system to run out of threads.
The first thing we can see in the thread dump is that there are no free application threads. And a quick look at the threads will show that most of them are doing exactly the same since some time ago, and are locked by "SAPEngine_Application_Thread[impl:3]_25":
If we make click on this thread, the information about how many threads are locked is shown. In that case, 135 of 150...
As we learn with the example 2, we could try to increase the number of application threads playing with the MaxThreadCount parameter. But 150 is a good number of threads for a portal, so this does not seem to be the problem. The thread seems to be doing some webdynpro stuff, so there is a good chance that multiple concurrent requests when running webdynpro applications are causing this deadlock situation. In that case, the first think we should do is to ensure that we have applied the latest webdynpro patch for our support package level. There is a SAP note dedicated to every released support package, and they should be our starting point.
Example 4: A lazy system...
As explained in SAP note 885645, "by default, the cluster lazy communication system has a permanently running thread, this is the lazy communication auto-close thread. This thread cares that the lazy connections between the nodes of the cluster are closed on time. This is done for resource (sockets and threads) saving reasons and is an optimization for clusters with more than 20-25 nodes."
If the lazy communication is active in our system, we will find two kind of threads in our dumps:
As there is a chance that the servers stop responding due to this lazy communication feature, we should try to stop the lazy communication to check if it is the root cause. The explanation on how to disable the lazy communication autoclose thread can be found at SAP Note 885645. This type of problem should only exist when there is more than one server node, anyway.
The end...
Well, I hope this little review of the thread dumps world is useful for you. Of course, there are lots of different situations which can happen in our systems, so we cannot expect to find all the answers on these four little examples. Anyway, this should help us to start the root cause analysis.