During Upgrade using SUM tool, stuck at Phase: MAIN_UPTRANS/STARTSAP_PUPG

[former_member230159]

Hello all,

I recently came across issue while performing upgrade.I was facing issue in the step : MAIN_UPTRANS/STARTSAP_PUPG.

There are a number of phases which are essential and need to be executed by SUM after STARTSAP_PUPG in the Post Processing roadmap before the system is unlocked for productive usage, and as such at this step, the system is not ready to be used normally.

In my case my upgrade was stuck at 89% and it was not moving forward.

The error screenshot from the SUM tool :

 



 

The error is :

Last error code set :

Process /usr/sap/SID/DVEBMGSXX/exe/sapcontrol  exited with return code 2  see 

/usr/sap/SID/SUM/abap/log/SAPup.ECO  for details 

System start failed

Validating the profile failed , see INSTANCE.OUT

 

My next step was to check if any help is available online ( in SAP help link / SAP wiki / SAP KBA )..

I could find some very nice written SAP wiki ( LINK , and LINK,)

But I was unable to find the solution 😐

So my next step was to go the " find information in the traces and logs ".

 

I started out by checking the very basic logs under /usr/sap/SID/abap/log [the first step to start checking the logs!!]

I started by checking the log SAPup_troubleticket log and I could find following entries :

This trouble ticket was created by SAPup on <DATE>
-------------------------------------------------------------
SAPup broke during phase STARTSAP_PUPG in module MAIN_UPTRANS / Transition to uptime
Error Message: Last error code set:
Process /usr/sap/SID/DVEBMGS00/exe/sapcontrol exited with 2, see
'/usr/sap/SID/SUM/abap/log/SAPup.ECO' for details
System start failed
Earlier, validating the profile failed, see INSTANCE.OUT

 

My next step was to check in SAPup.ECO log :

 

---> SAPup.ECO log

 

SAPup> Starting subprocess with PID <number> in phase 'INSTANCELIST_PRE' at <DATE>

EXECUTING /usr/sap/SID/DVEBMGS00/exe/sapcontrol -format script -prot NI_HTTP -host sapvmqas -nr 00 -function GetProcessList

14.08.2018 13:08:55
GetProcessList
OK
0 name: msg_server
0 description: MessageServer
0 dispstatus: GRAY
0 textstatus: Stopped
0 starttime:
0 elapsedtime:
0 pid: 3663
1 name: enserver
1 description: EnqueueServer
1 dispstatus: GRAY
1 textstatus: Stopped
1 starttime:
1 elapsedtime:
1 pid: 3664
2 name: disp+work
2 description: Dispatcher
2 dispstatus: GREEN
2 textstatus: Running
2 starttime: 2018 08 14 13:03:49
2 elapsedtime: 0:05:06
2 pid: 3665
3 name: gwrd
3 description: Gateway
3 dispstatus: GREEN
3 textstatus: Running
3 starttime: 2018 08 14 13:03:50
3 elapsedtime: 0:05:05
3 pid: 3668
4 name: icman
4 description: ICM
4 dispstatus: GREEN
4 textstatus: Running
4 starttime: 2018 08 14 13:03:50
4 elapsedtime: 0:05:05
4 pid: 3669

 

 

Here I could see the MessageServer and EnqueueServer was in GRAY  state. Meaning they were stopped. On this my first clue was to check the "processlist" output to see what is the output reported by sapcontrol output.

The output was :

 

[1]sapcontrol -nr 00 -function GetProcessList

16.08.2018 13:15:16
GetProcessList
OK
name, description, dispstatus, textstatus, starttime, elapsedtime, pid
msg_server, MessageServer, GRAY, Stopped, , , 24902
$(_EN), , GRAY, Stopped, , , 24903
disp+work, Dispatcher, GREEN, Running, 2018 08 14 16:50:11, 44:25:04, 24904
gwrd, Gateway, GREEN, Running, 2018 08 14 16:50:13, 44:25:02, 24905
icman, ICM, GREEN, Running, 2018 08 14 16:50:13, 44:25:02, 24906

 

So I logged into my system (Dispatcher was GREEN --> able to login 😉 )

 

When I logged into system and checked in SM12 I could see my enqueue server is running as a standalone engine !!!

 

---> Output from SM12 :

 

Lock management operation mode
Lock management in a standalone server
HOST: <HOSTNAME>
INST: 01

 

But then why was GetProcessList showing output having mes_server and enqueue server in it?

 

"There is something strange going on here "

So I again checked in system and this time I knew what had to be done....

 

I tried sapcontrol..GetProcessList  with the instance number 01 [as 01 instance is the instance number for ASCS ]

 

and the output was

 

[2]sapcontrol -nr 01 -function GetProcessList

16.08.2018 13:15:57
GetProcessList
OK
name, description, dispstatus, textstatus, starttime, elapsedtime, pid
msg_server, MessageServer, GREEN, Running, 2018 08 14 16:49:58, 44:25:59, 23382
enserver, EnqueueServer, GREEN, Running, 2018 08 14 16:49:58, 44:25:59, 23383

 

This is totally confusing!!    😞

 

On one hand I can see SUM tool is reporting error because it can see MessageServer and EnqueueServer are in GRAY status. But here I can see both are in GREEN status ( perfectly normal state )....

A idea hit me at this point in time... Why not check in the profile file for 00 instance where the issue was occurring ( or at-least being reported by SUM tool )...

"I had hit a jackpot here" 🙂

There was something terribly wrong in the profile for instance 00!!

 

There were entries in profile for 00 instance having :

 

Directory: /usr/sap/SID/SYS/profile
Name: <SID>_DVEBMGS00_<HOSTNAME>

# Entries for support of High-Availibitity release 740
Execute_07 = local rm -f $(_MS)
# Entries for support of High-Availibitity release 740
Execute_08 = local ln -s -f $(DIR_EXECUTABLE)$(DIR_SEP)msg_server$(FT_EXE) $(_MS)
# Entries for stand-alone enqueue server
Execute_09 = local rm -f $(_EN)
# Entries for stand-alone enqueue server
Execute_10 = local ln -s -f $(DIR_EXECUTABLE)$(DIR_SEP)enserver$(FT_EXE) $(_EN)
# Entries for support of High-Availibitity release 740
Start_Program_00 = local $(_MS) pf=$(DIR_PROFILE)$(DIR_SEP)QAS_DVEBMGS00_sapvmqas
# Entries for stand-alone enqueue server
Start_Program_01 = local $(_EN) pf=$(DIR_PROFILE)$(DIR_SEP)QAS_DVEBMGS00_sapvmqas
_MS ms.sapQAS_DVEBMGS00

 

Now these entries are not required at all .. when you are having a standalone engine having enqueue and message services running.

So my final resolution was to remove the above highlighted entries and then restart the SAP system.

 

Post this my upgrade resumed and it passed the 89% mark 🙂

 

My takeaways from this :-

When you have configured standalone enqueue server , there is no need to maintain the enqueue and message server related entries in the instance profile. These entries must be in the ASCS profile only.

 

 

UPDATE 2:

 

I recently was working a case where customer faced a similar issue but this time during perform installation.

 

They were getting this error while running SWPM.

 

INFO       2018-08-21 21:34:27.051 (root/sapinst) (startInstallation) [iaxxbjsmod.cpp:83] id=modlib.jslib.childAppReturn
Execution of the command "/usr/sap/SID/ASCS01/exe/sapcontrol -prot NI_HTTP -nr 00 -function GetProcessList" finished with return code 0. Output:

21.08.2018 21:34:27
GetProcessList
OK
name, description, dispstatus, textstatus, starttime, elapsedtime, pid
msg_server, MessageServer, GRAY, Stopped, , , 2797
disp+work, Dispatcher, GREEN, Running, 2018 08 21 21:32:28, 0:01:59, 2798
igswd_mt, IGS Watchdog, GREEN, Running, 2018 08 21 21:32:28, 0:01:59, 2799
gwrd, Gateway, GREEN, Running, 2018 08 21 21:32:29, 0:01:58, 2807
icman, ICM, GREEN, Running, 2018 08 21 21:32:29, 0:01:58, 2808

 

And the error on SWPM screen was :

<html> <head> </head> <body> <p> An error occurred while processing option <i>SAP Business Suite powered by SAP HANA > EHP2 for SAP CRM 7.0 On HANA > SAP HANA Database > System Copy > Target System > Standard System > Based on AS ABAP > Refresh or Move Database Instance(Last error reported by the step: Instance SID/DVEBMGS00 reached state SHUTDOWN after having state ACTIVE. Giving up)</i>. You can now: </p> <ul> <li> Choose <i>Retry</i><br>to repeat the current step. </li> <li> Choose <i>Log Files</i><br>to get more information about the error. </li> <li> Stop the option and continue later. </li> </ul> <p> Log files are written to /tmp/sapinst_instdir/BSHANA10/BSONHANA10/CRM/HDB/COPY/SYSTEM/STD/AS-ABAP/REF. </p> </body> </html>

 

Notice again the ASCS is running on ' 01 ' instance but when GetProcessList was queries for 00 instance it showed message server as GRAY .

The reason was entries related Start_XX , Restart_XX , Execute_XX was present in the DVEBMG00.

When the entries were removed from DVEBMGS00 profile and restart was taken.

 

Post this when SWPM was re run the issue had disappeared 🙂 .

 

 

I hope this information helps! 🙂

 

Kind regards,

Manjunath Hanmantgad