Issue :
Upgrade stuck in phase ' MAIN_SHDPREPUT/START_SHDI_PREPUT ' during SPS upgarde using SUM 1.0 from SPS 13 to SPS 21 for SAP Netweaver 7.4.
OS : Linux ( RHEL 6.5 )
database : Oracle 12c
Troubleshooting :
In the error screenshot you will be presented with the logs and traces which will indicate the location of the error logs.
Main logs to check would be SAPup.ECO ,'STARTSSC.LOG' and 'DEVTRACE.LOG'
The location of these files would be /usr/sap/SID/SUM/abap/log/SAPup.ECO OR /usr/sap/SID/SUM/abap/log/STARTSSC.LOG or /usr/sap/SID/SUM/abap/log/DEVTRACE.LOG
Part I
Upon checking the traces I can see the entries as in DEVTRACE.LOG
#!--------------------------------------------------------------
#!---- MASKING file
#!---- TIME:
PHASE: MAIN_SHDPREPUT/START_SHDI_PREPUT
#!--------------------------------------------------------------
1 ETQ201 Entering upgrade-phase "MAIN_SHDPREPUT/START_SHDI_PREPUT" ("")
2 ETQ367 Connect variables are set for standard instance access
4 ETQ399 System-nr = '00', GwService = 'sapgw00' Client = '000'
4 ETQ399 Environment variables:
4 ETQ399 dbs_ora_schema=SAPSR3
1 ETQ200 Executing actual phase 'MAIN_SHDPREPUT/START_SHDI_PREPUT'.
1 ETQ399 Phase arguments:
2 ETQ399 Arg[0] = 'START_SERVICE'
2 ETQ399 Arg[1] = 'DEVTRACE.LOG'
2 ETQ399 Starting shadow instance
4 ETQ399 Set database connect to shadow.
4 ETQ399 Set environment for shadow connect:
4 ETQ399 Set RFC variables for shadow connect:
4 ETQ399 System-nr = '02', GwService = 'sapgw02' Client = '000'
4 ETQ380 computing toolpath for request "TP_SHADOW_CONNECT"
4 ETQ381 request "TP_SHADOW_CONNECT" means "tp needs to connect to shadow system"
4 ETQ382 translates to group "R3UP_TOOL_GROUP_NEW"
4 ETQ399 Requirement for tp maps to 'exe'
4 ETQ383 translates to path /usr/sap/SID/SUM/abap/exe*
4 ETQ399 Set tool parameters for shadow connect with TPPARAM 'SHADOW.TPP'.
4 ETQ399 Set RFC variables for shadow connect:
4 ETQ399 System-nr = '02', GwService = 'sapgw02' Client = '000'
1 ETQ359 RFC Login to: System="<SID>", AsHost="HOSTNAME" Nr="02", Client="000", GwHost="HOSTNAME", GwService="sapgw02"
2EETQ231 RFC Login failed
2 ETQ353 Starting system
1 ETQ399 SYSTEM MANAGER: Instance list contains type: CI_YES.
1 ETQ399 SYSTEM MANAGER: instance found.
1 ETQ399 SYSTEM MANAGER: STARTSAP on Windows.
1 ETQ399 SYSTEM MANAGER: STARTSAP with SAPCONTROL.
1 ETQ399 SYSTEM MANAGER: Prestart on Windows.
1 ETQ399 SYSTEM MANAGER: StartService on Windows (this is actually a restart).
1 ETQ399 SYSTEM MANAGER: Getting state of service: HOSTNAME_02.
1 ETQ399 SYSTEM MANAGER: Restarting the service.
1 ETQ399 Make sure the service could reread profiles.
1 ETQ399 SYSTEM MANAGER: Calling getInstanceByNumberHost. Instance Number: 02, Instance Host: HOSTNAME.
1 ETQ399 SYSTEM MANAGER: Instance list contains number: 02, host: HOSTNAME
1 ETQ399 SYSTEM MANAGER: instance found.
1 ETQ399 SYSTEM MANAGER: ControlInstance with SAPCONTROL action PARAMETER VALUE for instance 02 and host HOSTNAME.
1 ETQ399 SYSTEM HEALTH MANAGER: running preCheck for instance 02 on host HOSTNAME.
2 ETQ399 SYSTEM HEALTH MANAGER: Checking instance number 2 for used ports on host 'HOSTNAME'.
3 ETQ399 SYSTEM HEALTH MANAGER: Port 3202 appears to be unused.
3 ETQ399 SYSTEM HEALTH MANAGER: Port 3302 appears to be unused.
3 ETQ399 SYSTEM HEALTH MANAGER: Port 3602 appears to be unused.
3 ETQ399 SYSTEM HEALTH MANAGER: Port 3902 appears to be unused.
3 ETQ399 SYSTEM HEALTH MANAGER: Port 8102 appears to be unused.
3 ETQ399 SYSTEM HEALTH MANAGER: Port 30201 appears to be unused.
3 ETQ399 SYSTEM HEALTH MANAGER: Port 50219 appears to be unused.
1 ETQ399 SYSTEM HEALTH MANAGER: call test rfc.
4 ETQ399 Set RFC variables for shadow connect:
4 ETQ399 System-nr = '02', GwService = 'sapgw02' Client = '000'
1 ETQ359 RFC Login to: System="<SID>", AsHost="HOSTNAME" Nr="02", Client="000", GwHost="HOSTNAME", GwService="sapgw02"
2EETQ231 RFC Login failed
1 ETQ399 SYSTEM HEALTH MANAGER: check for instance processlist.
1 ETQ399 SAPCONTROL MANAGER: getProcessList with host: HOSTNAME and instance: 02
3 ETQ120 TIME/DATE: PID <PID> execute '~\exe\sapcontrol.exe -format script -prot PIPE -host HOSTNAME -nr 02 -function GetProcessList', output written to '/usr/sap/SID/SUM/abap/log/SAPup.ECO'.
3WETQ123 TIME/DATE: PID <PID> exited with status 4 (time 0.00 real)
1 ETQ399 SAPCONTROL MANAGER: get parameter value with host: HOSTNAME, instance: 02.
3 ETQ120 TIME/DATE: PID <PID> execute '~\exe\sapcontrol.exe -prot PIPE -host HOSTNAME -nr 02 -function ParameterValue SAPSYSTEMNAME', output written to '/usr/sap/SID/SUM/abap/log/SAPup.ECO'.
3 ETQ123 TIME/DATE: PID <PID> exited with status 0 (time 0.00 real)
1 ETQ399 SYSTEM HEALTH MANAGER: running postCheck for instance 02 on host HOSTNAME
4 ETQ399 Set RFC variables for shadow connect:
4 ETQ399 System-nr = '02', GwService = 'sapgw02' Client = '000'
1 ETQ359 RFC Login to: System="<SID>", AsHost="HOSTNAME" Nr="02", Client="000", GwHost="HOSTNAME", GwService="sapgw02"
2EETQ231 RFC Login failed
2 ETQ399 SYSTEM HEALTH MANAGER: Checking instance number 2 for used ports on host 'HOSTNAME'.
3 ETQ399 SYSTEM HEALTH MANAGER: Port 3202 appears to be unused.
3 ETQ399 SYSTEM HEALTH MANAGER: Port 3302 appears to be unused.
3 ETQ399 SYSTEM HEALTH MANAGER: Port 3602 appears to be unused.
3 ETQ399 SYSTEM HEALTH MANAGER: Port 3902 appears to be unused.
3 ETQ399 SYSTEM HEALTH MANAGER: Port 8102 appears to be unused.
3 ETQ399 SYSTEM HEALTH MANAGER: Port 30201 appears to be unused.
3 ETQ399 SYSTEM HEALTH MANAGER: Port 50219 appears to be unused.
1 ETQ399 SAPCONTROL MANAGER: restartService with kernelstate: 0
3 ETQ123 TIME/DATE: PID <PID> exited with status 0 (time 0.00 real)
1 ETQ399 SYSTEM MANAGER: Start all instances.
1 ETQ399 SYSTEM MANAGER: Calling getInstances.
1 ETQ399 SYSTEM MANAGER: found instance for action START : Instance Number: 02
1 ETQ399 SYSTEM MANAGER: ControlInstance with SAPCONTROL action START for instance 02 and host HOSTNAME.
1 ETQ399 SYSTEM HEALTH MANAGER: running preCheck for instance 02 on host HOSTNAME.
2 ETQ399 SYSTEM HEALTH MANAGER: Checking instance number 2 for used ports on host 'HOSTNAME'.
3 ETQ399 SYSTEM HEALTH MANAGER: Port 3202 appears to be unused.
3 ETQ399 SYSTEM HEALTH MANAGER: Port 3302 appears to be unused.
3 ETQ399 SYSTEM HEALTH MANAGER: Port 3602 appears to be unused.
3 ETQ399 SYSTEM HEALTH MANAGER: Port 3902 appears to be unused.
3 ETQ399 SYSTEM HEALTH MANAGER: Port 8102 appears to be unused.
3 ETQ399 SYSTEM HEALTH MANAGER: Port 30201 appears to be unused.
3 ETQ399 SYSTEM HEALTH MANAGER: Port 50219 appears to be unused.
1 ETQ399 SYSTEM HEALTH MANAGER: call test rfc.
4 ETQ399 Set RFC variables for shadow connect:
4 ETQ399 System-nr = '02', GwService = 'sapgw02' Client = '000'
1 ETQ359 RFC Login to: System="<SID>", AsHost="HOSTNAME" Nr="02", Client="000", GwHost="HOSTNAME", GwService="sapgw02"
2EETQ231 RFC Login failed
1 ETQ399 SYSTEM HEALTH MANAGER: check for instance processlist.
1 ETQ399 SAPCONTROL MANAGER: getProcessList with host: HOSTNAME and instance: 02
3WETQ123 20190523131359: PID 12248 exited with status 4 (time 0.00 real)
1 ETQ399 SYSTEM HEALTH MANAGER: System is down, go on with start action
1 ETQ399 SAPCONTROL MANAGER: StartSystem with host: HOSTNAME and instance: 02
1 ETQ399 SYSTEM MANAGER: SAPControl action START failed for instance 02 ('SAPCONTROL MANAGER: call (sapcontrol.exe) failed with return code '-1'
').
1 ETQ399 SYSTEM MANAGER: CheckSystemStatus.
1 ETQ399 SAPCONTROL MANAGER: getProcessList with host: HOSTNAME and instance: 02
3 ETQ123 TIME/DATE: PID <PID> exited with status 0 (time 0.00 real)
1EETQ399 SYSTEM MANAGER: START of mandatory instance 02 on server HOSTNAME has failed
2EETQ399 Starting shadow instance failed
1EETQ399 Last error code set is: Shadow instance
1EETQ399Xcouldn't be started, check 'STARTSSC.LOG' and 'DEVTRACE.LOG': Process W:\Stack18\SUM\SUM\abap\exe/sapcontrol.exe exited with 2, see '/usr/sap/SID/SUM/abap/log/SAPup.ECO' for details
1EETQ204 Upgrade phase "START_SHDI_PREPUT" aborted with severe errors ("TIME/DATE")
i repeated the Phase
Part II :
Next step is to check the dispatcher trace file. The reason is because the isnatnce is not starting , so the basic files to check are dev_disp,dev_w* traces.
In dev_disp I can see the entries as:
EgInit: EG initialized with 2192M in 1 segments
mm.dump: set global maximum dump mem to 192 MB
EsRegisterEmCheck: Register EmGetEsHandles at 14161e674
rdisp/calculateLoadAverage : 1
rdisp/snapshot(CREATE): DP_CS_RESOURCES_EXHAUSTED
rdisp/snapshot(PERIOD):300 sec
rdisp/snapshot(LINES):10000 lines
rdisp/snapshot(C-STACK):C-STACK = YES
*
** ERROR => invalid eyecatcher 0 for mtx=1108f710 userid [mtxxx.c 524]
*** ERROR => DpLogonCntTabSetResourceLimit: MtxLock failed (4) [dpLogonCntTa 1048]
***LOG Q0K=> DpMsAttach, mscon ( HOSTNAME) [dpMessageSer 1919]
Resolution :
Looking at the above entries I can see that the issue is with the
DpLogonCntTabSetResourceLimit.
The DpLogonCntTabSetResourceLimit is related to parameter rdisp/user_resource_limit (User specific resource limits ).
The parameter is very interesting parameter. This parameter is used to limiting Application Server ABAP Resources per User .It contains subparameters WARN and REJECT to specify the limits for resources used by a single user.
For more information you can find the details in link here :
Link
Finally to resolve the issue I could find a SAP note
2742661 - DP: Error in DpLogonCntTab if table size is 0 //
There is a workaround available as : Set the parameter "
rdisp/user_resource_limit = WARN=100, REJECT=100"
But to permanently resolve the issue I applied the SAP patch as mentioned in note
2742661.
Hope this helps.
Kind regards,
Manjunath Hanmantgad
