Identify the owner of the process holding the mutex lock

Purpose

To identify the process holding the mutex lock which will help in troubleshooting mutex related problems.There are different processes holding the mutex lock.In order to identify which is that process and how to release that held mutex we must first understand how to identify the process.

Breaking down the MtxLock

Firstly what is meaning of the MtxLock function ? By definition this means wait for a lock to become available. Mutexes can’t be recovered automatically if a process dies while still holding a lock.

The Mutex related trace entries

In the developer traces of the work processes  (dev_w*) we can find the entries as:

example #1

example #2

Now lets break down each example one by one.

From example #1

*** WARNING => MtxLock a00010020000270 RQ_Q_WOD owner=30004 deadlock ? [mtxxx.c 641]

Now here we can see the owner is : owner=30004

Similarly in example #2 we can see the owner is owner=30001 

Identifying the caller id aka client id aka CID

First I would like to list out the predefined client id’s in the kernel.

The Client id range starts from 30000 and is end to 32767.

By definition it states as ( part of internal SAP coding I cant provide more detail due to security reasons ) :

30000+0 = 30000 –> Owner is dispatcher

30000+1 = 30001 –> Owner is ICM

30000+2 = 30002 –> Owner is the java control process

30000+3 = 30003 –> Owner is dpmon process

30000+4 = 30004 –> Owner is gateway

30000+5 = 30005 –> Owner is VMC process

30000+6 = 30006 –> Owner is ES (extended segment) memory area

30000+7 = 30007 –> Owner is EM (extended memmory) memory area

30000+8 = 30008 –> Owner is EM (extended global) memory area

Troubleshooting the entries

Going one step further we now know who is the owner of the mutex.Our next step is checking the related developer trace file to rectify the issue.

From example #1

*** WARNING => MtxLock <hex address> RQ_Q_WOD owner=30004 deadlock ? [mtxxx.c 641]

Here we will check in the gateway trace (dev_rd) to see what is the issue.

From example #2

*** WARNING => MtxLock <hex address> MTXGLOB owner=30001 deadlock ? [mtxxx.c 2574]

Here we will check in the ICM (dev_icm) to see what is the issue.

TIP: Please also check the dev_*.old traces respectively.

Let me know in case any queries.

Kind regards,

Manjunath Hanmantgad