To identify the process holding the mutex lock which will help in troubleshooting mutex related problems.There are different processes holding the mutex lock.In order to identify which is that process and how to release that held mutex we must first understand how to identify the process.
Breaking down the MtxLock
Firstly what is meaning of the MtxLock function ? By definition this means wait for a lock to become available. Mutexes can’t be recovered automatically if a process dies while still holding a lock.
The Mutex related trace entries
In the developer traces of the work processes (dev_w*) we can find the entries as:
|*** WARNING => MtxLock <hex address> RQ_Q_WOD owner=30004 deadlock ? [mtxxx.c 641]|
|*** WARNING => MtxLock <hex address> MTXGLOB owner=30001 deadlock ? [mtxxx.c 2574]|
Now lets break down each example one by one.
From example #1
*** WARNING => MtxLock a00010020000270 RQ_Q_WOD owner=30004 deadlock ? [mtxxx.c 641]
Now here we can see the owner is : owner=30004
Similarly in example #2 we can see the owner is owner=30001
Identifying the caller id aka client id aka CID
First I would like to list out the predefined client id’s in the kernel.
The Client id range starts from 30000 and is end to 32767.
By definition it states as ( part of internal SAP coding I cant provide more detail due to security reasons ) :
30000+0 = 30000 –> Owner is dispatcher
30000+1 = 30001 –> Owner is ICM
30000+2 = 30002 –> Owner is the java control process
30000+3 = 30003 –> Owner is dpmon process
30000+4 = 30004 –> Owner is gateway
30000+5 = 30005 –> Owner is VMC process
30000+6 = 30006 –> Owner is ES (extended segment) memory area
30000+7 = 30007 –> Owner is EM (extended memmory) memory area
30000+8 = 30008 –> Owner is EM (extended global) memory area
Troubleshooting the entries
Going one step further we now know who is the owner of the mutex.Our next step is checking the related developer trace file to rectify the issue.
From example #1
*** WARNING => MtxLock <hex address> RQ_Q_WOD owner=30004 deadlock ? [mtxxx.c 641]
Here we will check in the gateway trace (dev_rd) to see what is the issue.
From example #2
*** WARNING => MtxLock <hex address> MTXGLOB owner=30001 deadlock ? [mtxxx.c 2574]
Here we will check in the ICM (dev_icm) to see what is the issue.
TIP: Please also check the dev_*.old traces respectively.
Let me know in case any queries.