Handle JAVA hung issue during Dual stack upgrade?
A typical problem that we encounter during the dual stack system upgrade is object locking while JAVA system try installing the java archives. The issue is commonly seen during the upgrade of Solution Manager 7.1 to Solution Manager 7.2 system while the system is in pre-processing and downtime phases. Java upgrade would go in a hung state while deploying the archives and do not move forward. This happens due to the enqueue locks created in the Database for parallel deployment.
You can find the traces of locks in deploy.X.trc or deploy_api.X.trc files under SUM directory /SUM//sdt/system/<SID>/<JAVA shadow instance>/work/ or /SUM//sdt/system/<SID>/<JAVA instance>/work/
Example:
Caused by: com.sap.engine.frame.core.configuration.ConfigurationLockedException: Cannot lock Configuration “cluster_config” because it is already locked.
#at com.sap.engine.core.configuration.impl.ServerEnvironment.lockConfiguration(ServerEnvironment.java:541)
#at com.sap.engine.core.configuration.impl.cache.ConfigurationController.openConfiguration(ConfigurationController.java:916)
#at com.sap.engine.core.configuration.impl.ConfigurationHandlerImpl.openConfiguration(ConfigurationHandlerImpl.java:1049)
Solution:
To counter this update the deploy_threads property to 1 in configtool as mentioned below and stop/start the instance using below commands:
sapcontrol -nr XX -function Stop
sapcontrol -nr XX -function Start
You can monitor the JAVA instance status using
sapcontrol -nr XX -function GetSystemInstanceList
GetSystemInstanceList
OK
hostname, instanceNr, httpPort, httpsPort, startPriority, features, dispstatus
XYZ, 4, 5XX13, 5XX14, 1, GATEWAY|MESSAGESERVER|ENQUE, GREEN
XYZ, 7, 5XX13, 5XX14, 2, ABAP|GATEWAY|MESSAGESERVER|ENQUE|ICMAN, GREEN
XYZ, 3, 5XX13, 5XX14, 3, J2EE|ICMAN, GREEN
Any status other than GREEN is not good, you will have to sun the Start command again.
Now this was all done in higher version of NetWeaver like 7.3 or 7.4 where the latest version on offline configtool but the real problem was missing service “tc~bl~deploy_controller” in lower version of configtool. Typical example is solution manager upgrade, during preprocessing the JAVA was still on NW7.02 and the above service was not present in configtool. So in this situation, just stop/start shadow instance services using below commands.
Stop the SCS and JAVA shadow instance using below commands:
/SUM/sdt/data/kernel/sapcontrol -nr XX -host <hostname>-user <SID>adm password -prot NI_HTTP -function Stop
Start sapstartsrv services for shawdow instance:
/SUM/sdt/data/kernel/sapstartsrv pf=/sapmnt/<SID>/SUM/sdt/system/<SID>/SYS/profile/<SCS shadow instance profile> -D
/SUM/sdt/data/kernel/sapstartsrv pf=/sapmnt/<SID>/SUM/sdt/system/<SID>/SYS/profile/<JAVA shadow instance profile> -D
Start SCS and JAVA shadow instance using:
/SUM/sdt/data/kernel/sapcontrol -nr XX -host <hostname> -user <SID>adm password -prot NI_HTTP -function Start
*XX = instance number
Ref:
1797346 – DEPLOY_ONLINE_SHD_DM,START_SHD_J2EE_DM or DEPLOY-SHADOW-INSTANCE: Start AS JAVA shadow instance manually
1884294 – Deployment fails due to a locking issue
2419202 – Software Update Manager (SUM) hangs in deployment in the step: DEPLOY-ONLINE-DB