Shutting down db2 luw fully when it is hung (has processes hung in unix / linux)

Shutting down db2 luw fully when it is hung (has processes hung in unix / linux)

NOTE:  one should collect troubleshooting data (db2support.zip, db2fodc -hang full{db2fodc will try to collect all data it can collect while the database is hanging. one should use this command only in the case of a database hang. one should not use it on a normally running system}, etc) first before doing these steps, if one wants root cause analysis of abnormal db2 stoppage to be done if and when db2 is restarted after these steps.

*********************************************************************************
if the following fails to stop DB2 LUW on unix / linux . . .
*********************************************************************************
db2 list applications
db2 force applications “(<pid>)”
db2 force applications “(####)”
db2 list applications
db2stop
db2stop force
*********************************************************************************

steps (brute force method) to shutdown DB2 LUW on unix / linux
*********************************************************************************
using . . . x6d . . . as the <SID> in the following items . . .
*********************************************************************************
1. db2 list applications / db2 force applications . . . / db2stop / db2stop force
…….if any of these hang / if any of these fail to finish . . .
2. db2_kill . . . this may or may not succeed . . .
…….if db2_kill hangs also just continue with the following steps
3. ps -ef | grep -i x6d . . . check for db2 processes run by db2x6d or x6dadm
4. if there is(are) orphaned process(es) still
…….kill them using unix . . . kill -9 <process_id>
5. ipclean -a
6. ipcs -a | grep -i x6d . . . make sure ipclean succeeded . . .
…….this command should come back with empty output
7. if . . . ipcs -a | grep -i x6d . . . does come back with output
…….ipclean failed to clear up all the IPC resources
one will have to delete them manually as the root userid using ipcrm . . .
ipcrm -q #### (for message queues)
ipcrm -m #### (for shared memory)
ipcrm -s #### (for semaphores)
*********************************************************************************
*********************************************************************************
Below is an example of the above commands done on an in-house sandbox SAP system
with <SID>=X6D.

0. prepare situation for the seven steps listed above
***********************************************************************************

0.1. logged in with putty session as db2<sid> userid.
***********************************************************************************
telnet (slingaix)
AIX Version 6
Copyright IBM Corporation, 1982, 2015.
login: db2x6d
db2x6d’s Password:
*******************************************************************************
* Welcome to AIX Version 6.1! *
*******************************************************************************
Last unsuccessful login: Mon May 16 20:26:18 2015 on
Last login: Fri May 20 02:18:56 2016 on

slingaix:db2x6d 1>
**********************************************************************************

0.2. verify csh shell is being used.
**********************************************************************************
slingaix:db2x6d 1> ps -p $$
PID TTY TIME CMD
22282816 pts/8 0:00 csh
slingaix:db2x6d 2>
**********************************************************************************

0.3. set prompt to expose shell:[userid]@[hostname]:[current_directory]>
**********************************************************************************
slingaix:db2x6d 2> setenv SHELL csh
slingaix:db2x6d 3> setenv xxshell
slingaix:db2x6d 3> set prompt=”`echo $xxshell`:`id |cut -f2 -d'(‘ |cut -f1 -d’)’`@`hostname | cut -f1 -d’.’`:`pwd`> ”
slingaix:db2x6d 3> alias idmw “id |cut -f2 -d'(‘ |cut -f1 -d’)'”
slingaix:db2x6d 4> alias cd ‘chdir \!* && set prompt=”`echo $xxshell`:`idmw`@`hostname | cut -f1 -d’.’`:`pwd`> “‘
csh:db2x6d@slingaix:/db2/db2x6d>
**********************************************************************************

0.4. list processes associated with sap and db2 luw . . .
(the list below was edited to only show db2 processes)
**********************************************************************************
csh:db2x6d@slingaix:/> ps -ef | grep sap | grep -v grep; ps -ef | grep db2 | grep -v grep
. . .
db2x6d 4653258 1 0 02:17:52 – 0:00 /db2/db2x6d/sqllib/bin/db2bp 11862174A820 5 A
. . .
db2x6d 24576090 22282816 0 02:34:07 pts/8 0:00 db2
db2x6d 24772774 22872206 0 02:37:19 pts/10 0:00 db2pd
. . .
root 1507824 1 0 02:13:44 – 0:00 db2wdog 0
db2x6d 4260152 1 0 02:17:03 – 0:00 /db2/db2x6d/sqllib/bin/db2bp 24838148A820 5 A
root 5636466 23528312 0 02:13:44 – 0:00 db2ckpwd 0
. . .
root 10289526 23528312 0 02:13:44 – 0:00 db2ckpwd 0
db2x6d 14942710 23528312 0 02:13:45 – 0:00 db2vend (PD Vendor Process – 258)
. . .
root 19595734 23528312 0 02:13:44 – 0:00 db2ckpwd 0
db2x6d 4391668 1507824 0 02:13:46 – 0:03 db2acd 0
db2x6d 4784718 1 0 02:21:04 – 0:00 /db2/db2x6d/sqllib/bin/db2bp 5505540A820 5 A
db2x6d 6161042 1 0 02:22:01 – 0:00 db2
db2x6d 13501126 1 0 02:34:07 – 0:00 /db2/db2x6d/sqllib/bin/db2bp 22282816A820 5 A
db2x6d 15008262 1 0 02:28:44 – 0:00 /db2/db2x6d/sqllib/bin/db2bp 19333540A820 5 A
. . .
db2x6d 4391864 4784340 0 02:25:22 pts/11 0:02 topas
db2x6d 11273024 11862174 0 02:23:15 pts/6 0:00 db2
. . .
db2x6d 18678590 1 0 02:17:30 – 0:00 /db2/db2x6d/sqllib/bin/db2bp 9240584A820 5 A
db2x6d 22348740 24838148 0 02:17:03 pts/2 0:00 db2
db2x6d 22741888 9240584 0 02:17:30 pts/5 0:00 db2
db2x6d 23528312 1507824 134 02:13:44 – 16:08 db2sysc 0
csh:db2x6d@slingaix:/>

db2 list applications / db2 force applications . . . / db2stop / db2stop force
if any of these hang / if any of these fail to finish . . .
***************************************************************************************
atttempt to execute . . . db2 list applications
***************************************************************************************
csh:db2x6d@slingaix:/> db2 list applications
. . . this hangs . . .
***************************************************************************************

1. opened new putty session . . .
***************************************************************************************
since session in 1. above is hung, then opened new putty session . . .
then did steps 0.1. thru 0.3. above

2. db2_kill . . . this may or may not succeed . . .
if db2_kill hangs also just continue with the following steps.
**********************************************************************************
csh:db2x6d@slingaix:/> /db2/db2x6d/sqllib/bin/db2_kill
ipclean: Removing DB2 engine and client’s IPC resources for db2x6d.
csh:db2x6d@slingaix:/>

3. ps -ef | grep -i x6d . . . check for db2 processes run by db2x6d or x6dadm
**********************************************************************************
csh:db2x6d@slingaix:/> ps -ef | grep -i x6d
. . .
db2x6d 24772774 22872206 0 02:37:19 pts/10 0:00 db2pd
. . .
db2x6d 5898870 1 0 02:44:11 – 0:00 /db2/db2x6d/sqllib/bin/db2bp 11862174A820 5 A
. . .
db2x6d 22348740 24838148 0 02:17:03 pts/2 0:00 db2
db2x6d 23528330 1 0 02:44:21 – 0:00 /db2/db2x6d/sqllib/bin/db2bp 19333540A820 5 A
csh:db2x6d@slingaix:/>

4. if there is(are) orphaned process(es) still, kill them using unix . . . kill -9 <process_id>
***************************************************************************************
csh:db2x6d@slingaix:/> kill -9 24772774 5898870 22348740 23528330
csh:db2x6d@slingaix:/>
************************************************************************************

4.1 verity the kill -9 succeeded
************************************************************************************
csh:db2x6d@slingaix:/> ps -ef | grep -i x6d
. . .
csh:db2x6d@slingaix:/>

5. ipclean -a
***************************************************************************************
csh:db2x6d@slingaix:/> which ipclean
/db2/db2x6d/sqllib/bin/ipclean
csh:db2x6d@slingaix:/> ipclean -a
ipclean: Removing all IPC resources for db2x6d.
csh:db2x6d@slingaix:/>

6. ipcs -a | grep -i x6d . . . make sure ipclean succeeded . . .
…….this should come back with no output
***************************************************************************************
csh:db2x6d@slingaix:/> ipcs -a | grep -i x6d
m 2097169 0x00002752 –rw-r—– x6dadm sapsys x6dadm sapsys 0 348685544 2950134 2950134 …
m 1048693 0x0382be85 –rw-r—– x6dadm sapsys x6dadm sapsys 0 4096 4326170 4391668 …
m 4194423 0x0000271a –rw-r—– x6dadm sapsys x6dadm sapsys 0 200000000 2950134 2950134 …
s 133 0x002deeeb –ra-r—– x6dadm sapsys x6dadm sapsys 60 22:25:50 6:11:51
s 142 0x002deeea –ra-r—– x6dadm sapsys x6dadm sapsys 60 22:07:25 6:12:06

7. if . . . ipcs -a | grep -i x6d . . . does come back with output
ipclean failed to clear up all the IPC resources
one will have to delete them manually as the root userid using ipcrm . . .
ipcrm -q #### (for message queues)
ipcrm -m #### (for shared memory)
ipcrm -s #### (for semaphores)

one with the ability to login in as root must complete the last items . . .
***************************************************************************************
csh:root@slingaix:/> ipcrm -m 2097169
csh:root@slingaix:/> ipcrm -m 1048693
csh:root@slingaix:/> ipcrm -m 4194423
csh:root@slingaix:/> ipcrm -s 133
csh:root@slingaix:/> ipcrm -s 142