How the dual stack split killed the HA-scenario
Introduction
The upgrade of our BI 7.01 (ABAP+JAVA Dualstack) was requested by our BW-team. They wanted
to use the new features of NW 7.3. The better integration of the BO tools was
also a benefit of an upgrade. The upgrade was divided into two parts:
Part 1: The Dual Stack Split.
This procedure separates the JAVA stack from the ABAP stack. Before the dual stack split the JAVA and ABAP runs under one SID. After the split you have two SIDs (the old SID for ABAP and one new SID for JAVA).
Part 2: The Upgrade
This was an usual upgrade.
The Problem
Before the split we had this topology:
After the split and the upgrade the HA features didn’t work anymore 🙁 In a dualstack system the sap gateway service is addressed by sapgw$$. $$ is substituted by the actual system number.
The following picture shows the topology after the split:
The problem is the Gateway service. sapgw00 is configured in several destinations, in the UME and the JCO.
It was quite plain to me. The gateway service is HA critical!
A little shame on SAP
This is what you find about “high availability” in the documentation of the Dual stack split:
In the installation documentation of a java stack you find the HA configuration of a “normal” java stack! But what is the difference between a “normal” java stack and a split java stack? It is the UME! In a split java stack the UME is in ABAP! I’m very glad to know that nobody heard my comments when doing the HA cluster test on our production system!!!
In this picture you see the topology of a HA java stack (found in the installation guide). But what is to do, when using the java stack in a BI system with HA? I know somewhere in the paper jungle of SAP you can find something. But sap gives no hint in the standard documentation 🙁
The solution
The following configuration needs to be changed:
1. separate the gateway service
2. logon groups in ABAP
3. the UME Backend Connection
4. JCo RFC-Provider
5. SLD
Separate the gateway service. OSS note 1010990 ” Configuring a Standalone Gateway in an HA ASCS instance” explains how to configure the gateway service.
Step 1:
*Do this procedure for each scs.lst of each Instance.
*Changes to the instance-exe-directory. Example for Instance is ASCS00 or DVEBMGS02…
sidadm> cd /usr/sap/<SID>/<Instance>/exe
*edit the sapcpe config file
sidadm> vi scs.lst
*add this line to the file
gwrd
Step 2:
*Changes to the profile-directory
sidadm> cdpro
*gives you the path <pro_path> of profile-dir.
sidadm> pwd
*edit ASCS instance profile
sidadm> vi <pro_path>/<SID>_ASCS<Sys.Nr.>_<hostname>
*add this line to the profile
gw/netstat_once = 0
Step 3:
*edit the start profile of ASCS
sidadm> vi <pro_path>/START_ASCS<Sys.Nr.>_<hostname>
*Search the first Start_Program_<xx>-entry in the profil, then enter after that line:
_GW=gwrd$(FT_EXE)
*search the Start_Program_<xx>-entry with the highest value of <xx>, then enter:
Start_Program_<xx+1> = local $(DIR_EXECUTABLE)/$(_GW) pf=$(DIR_PROFILE)/<SID>_ASCS<Sys.Nr.>_<hostname> -no_abap
Step 4:
*check in /etc/services the entries:
sapgw<Sys.Nr.>
sapgw<Sys.Nr.>s
restart system
The new gateway service is sapgw<Sys.Nr.> because of the system number of ASCS.
logon groups in ABAP
Logon to the ABAP system and enter transaction SMLT. Configure for each Instance a logon group (in the picture it is PUBLIC)
UME Backend Connection
Comment
You can determine the hostname of
– message server (1)
– gateway server (2)
(1) The name of the Instance Profile of ASCS is <SID>_ASCS<Nr.>_<hostname>
(2) Same hostname as hostname of ASCS
http://<hostname>:50<Sys.Nr. of java>0/nwa
->Configuration
->Infrastructure
->Destinations
JCo RFC-Provider
http://<hostname>:50<Sys.Nr. of java>0/nwa
->Configuration
->Infrastructure
->Destinations
http://<hostname>:50<Sys.Nr. of java>0/nwa
->Configuration
->Infrastructure
->Jco RFC Provider
Login to the ABAP BI system. Go to Transaction SE16 -table-> RSPOR_T_PORTAL
In field RFCDEST you can find the JCo RFC-Destination in ABAP. Go to transaction SM59 and find the RFC destination.
Enter the gateway configuration
SLD
http://<hostname>:50<Sys.Nr. of java>0/sld
->Administration
->Settings
Logon to ABAP BI system and go to transaction SM59. Enter the gateway configuration (like in RFC destination of JCo) in the following RFC destinations:
EPILOG
Be aware: You get no hint in the documentation of the dual stack split or the installation about the correct HA settings.
Best regards
Willi Eimler
Hello Willi,
Nice and interesting blog !
I had the same kind of problem last year in a simpler case, when upgrading an HA ECC6 system from 4.7 to ECC6 EHP5.
We had external systems connected to our system as a registered RFC server in the gateway of the central instance of R/3 4.7. As the CI was a cluster resource, this was not a problem. But after upgrading to ECC6, the CI disappeared and only the ASCS was a cluster resource.
Like you I discovered that SAP forgot that the gateway could be a SPOF and needed to be a cluster resource. I found the same note as you and the solution was also to start a gateway process in the ASCS.
I agree with you that this should be the standard configuration and we should not have to discover the information hidden in a SAP note....
Best Regards,
Olivier
Hi Olivier,
thanx for your comment!
In my opinion the problems with the standard documentation results in not testing the
procedures! No HA environment is functional after a dual stack split!
Best regards
Willi Eimler
Hi Willi,
Thanks once again for this intersting blog! Although I understand your frustration, I doubt that your statement is really correct for all cases ("no HA environment"), as I am pretty sure that regular automated tests are executed also for HA dual-stack split scenarios (I heard at least of tests for Microsoft Windows, but would also have to cross-check again with the porting teams - but maybe we should consider to extend those tests 🙂 ).
Anyhow, the Java stack gets set up newly as part of the dual-stack split procedure and as a consequence, it is good advice to re-check the corresponding configuration and adapt it like after a fresh installation of a new HA system or after a system copy, as the Java stack got a new SAP system ID. This and the fact that the UM configuration is not changed (but remains in the ABAP system) is described in the dual-stack split guide (sections "3.2 User Management" and "6.1 Follow-Up Activities Checklist", pointing to the system copy guide).
But I totally agree that this is not straight-forward, especially for HA systems. As the corresponding procedures lie in the responsibility of our hardware partners, I plan to approach them with your feedback and hopefully, we can improve the procedure together in the right direction. Could you please let me know your exact platform (OS/DB)?
Thanks once again for sharing your experience, your support and your understanding,
Boris
Hi Boris,
sorry for the late reply. I was on a short holyday (i need to get rid of my "leave days";)!
We are using serviceguard with IA64 HP-UX 11.31 and Oracle 11.2.0.3.
Thanx for your reply!
Best regrads
Willi Eimler