Refreshing HANA systems involving XSA tenants

XSA specific HANA DB refresh steps:

In general, HANA refreshes are pretty straight forward and easy,.However, that is not the case when you have XS advanced tenants involved in your landscape. Many additional steps must be performed to bring back the target system to be fully operational with XSA applications without any issues.

This blog covers the pre-steps,refresh and post-steps specific to XSA associated HANA DB.

Note: This blog assumes that you are well aware of normal HANA refreshes and does not cover the normal HANA refresh related details. This documents assumes that the XSA is installed in separate tenant rather than on SystemDB .

Sample system:

In this scenario we are going to refresh DWE with QWE,DER with QER and DXS with QXS . We will not be refreshing SystemDB as we are only considering the scenario where XSA is installed in tenant DB -QXS/DXS . This tenant QXS/DXS is called the meta data tenant of XSA

The same can be verified by logging into OS level using sidadm and executing command

XSA list-tenants

To explain above screen shot, XSA will have 1 org unit and multiple spaces.Each spaces will be associated with a tenant . Here I have represented org unit as ORG across the tenants and spaces as SPACE1 which is the space of tenant DWE and SPACE2 which is the space of tenant DER.

The tenant DXS is the tenant on which we have installed XSA and holds the meta data of the XSA persistence . This can be distinguished by the entry highlighted above which reads as “XS Advanced platform persistence : YES” and will be having the same org unit in a landscape (ORG) and SAP default space “SAP”

Prerequisites with regards to XSA tenant enabled system refresh:

1.Individual tenants must not be refreshed as a best practice . If a source system consists of 3,XSA tenants which includes 1 with XSA meta data, the target must also have same number of XSA tenants .

2.SAP recommends to perform as PIT refresh for all the XSA tenants in one SystemDB.

3.XSA will be disabled at the start of the refresh as a first step

4.You know the password for XSA_ADMIN which is a master user in XSA

A.Pre-refresh steps:

1.Identify the XSA cockpit URL

Login to XSA meta data tenant and inside SAP space ,identify the XSA cockpit URL if not knownAs sidadm,

xs login –o ORG -s SAP

Once inside the SAP space, type

xs apps

From the above output identify the XSA-cockpit URL.

2.Open the URL in browser and take the pre-snaps of important data

A.Spaces:

Click on each spaces and record the screen shot of applications,services,User provided Services,Routes etcWe can get these details even from OS level by logging into each space and collect information

           -To login to each space xs login-o <ORG> -s <SPACE1>

– To capture all the application running in the space -> xs apps

-To capture the services in each space -> xs services

-To capture the routes in each space -> xs routes

-To capture the list of MTARs deployed on each space -> xs mats

B.Users

In our case post refresh we will have users only from source and hence we record the list of users in XSA to be recreated post refresh.

-To find the list of XSA users, goto XSA cockpit and click on user management

C.Users and space mapping:

Each of the users above will have access to individual space only if there is an explicit  mapping of Space roles like  SPACE DEVELOPER,SPACE MANAGER etc  to the users in the respective spaces.

Click on each space from XSA cockpit home  page and go to members and capture the screen shot for each space members, so that same can be validated and recreated post refresh

D.Current custom role collections.

Goto cockpit home page and click on Role Collections .

If there are some custom role collections that is not present in source, that screen shot can be captured . Click on each role collections to capture the screen shot of associated roles inside each role collections.

E. Exporting trust configuration:

If you have AD connected to the XSA for SSO , then we can download the meta data of original system and can be reimported again post refresh

F.Take screenshots of configuration file:

From System DB take a screen shot of below in case of SSO configuration in place

G.Take a note of api URL in target

As sidadm execute the command xs api

<sid>adm@sddgfdgg:/usr/sap/<SID>/SYS/global/hdb/custom>xs api

API endpoint: https://<host>:<port>

SSL trust: The authenticity of host ‘xx.cdscdsf.com’ is established by the ‘/hana/shared/<SID>/xs/controller_data/controller/ssl-pub/router/default.root.crt.pem’ certificate.

H.Copy entries of UPS (User provided services)

Post refresh, as all the UPS must be brought back with original configuration as it will hold production data post refresh. Hence  click on UPS entries and take screen shot of each of the entry

I. Make a note of general health check on XSA

As sidadm , execute XSA diagnose command

J.Ensure source and target with respect to xsa_sizing parameter is same to avoid memory issue or XSA instability issues post refresh

K.Login to each space and take screen shot and make a note of API url with below command:

xs login –o ORG –s SPACE1

xs login –o ORG –s SPACE2

xs login –o ORG –s SAP

B. Refresh:

1.Before starting with refresh disable the XSA services

As sidadm, execute XSA disable

After this all the xsa related services will be down.

2.Take a backup of the File-System Service (FSS) in an XS advanced backup operation, run the backup-fss command as <sid>adm,

XSA backup-fss

3.Perform a PIT restore from source to target w.r.t to the decided PIT on  all the target tenants  respectively using normal HANA restore method.

QWE will be restored with the hh:mm:ss to DWE

QER will be restored with the hh:mm:ss to DER

QXS will be restored with the hh:mm:ss to DXS

Post successful completion, your XSA related services will still be disabled.

4.After completing the restore successfully , disable the XSA services if found active.

Execute command XSA disable as sidadm

5.Now point the XS advanced platform services to the tenant database containing the XS advanced platform data we just restored, for example, by running the following command as <sid>adm user, execute below command

XSA select-xsa-runtime-db DXS

This command also restores the fss backup that we had took in step 2, XSA backup-fss along with enabling the XSA services that were red w.r.t screen shot shown in step 1.All services in systemdb will be green by now.

6.Run XSA health check and make a note of all the errors if any .Ensure all the XSA tenants are intact . Update the tenant password if needed.

XSA diagnose

7.Rename the XSA tenant space to original target

After refresh, we see that the spaces that are connected to the target is similar to source system and hence we need to rename the space.

You can check the same by executing

 XSA list-tenants

Now by referring to earlier details of source and target spaces, rename the target spaces which will be using the source space name , to original target

As sidadm, execute the below command

xs rename-space SPACEK SPACE1

xs rename-space SPACEY SPACE2

Now rerun XSA list-tenants to check the tenants, and their space mapping .

To display the spaces alone you can uses command xs spaces

 8.Reapply SSL certificate as the XSA cockpit URL will not show insecure if it was secured earlier.

As sidadm execute below. (Refer last prestep f for the URL)

xs l -a https://<fdsf.dfsdfsf.dgsgfs.net>:nnnnn–skip-ssl-validation

then,

cccadm@sfsfasfsaf:/hana/shared/CCC/HDB00/sapghercsw/sec>openssl pkcs12 -in star.dfsdfggfgg net.p12 -out star.sdasfsdfgdsg.pem -nodes

Enter Import Password:

MAC verified OK

Now,Set the certificate

xs set-certificate <URL.net> -k pkey -c chain

Now restart XSA

As sidadm, execute the command -> XSA restart

9.Login to each space and perform a general check if any

xs login –o ORG –s SPACE1

xs login –o ORG –s SPACE2

xs login –o ORG –s SAP

10.Now login to XSA with the API endpoint URL that we get from previous step

As sidadm

xs l -a https://>fdsf.dfsdfsf.dgsgfs.net>:nnnnn–skip-ssl-validation

11.Based on the screen shots taken during prerefresh update below in xsa cockpit URL

-UPS (User provided services)

-Users

-Space user mapping

-Role and role collection mapping

-Reimport the meta data and ensure that the entity id matches based on pre-step e =>login.entityid

12.Rebind all the xs application in each space using the command below .This will ensure that all the production environment variable in applications will get updated

Login to each space and execute below command

xs login –o ORG –s SPACE1

xs rebind-services hana  or xs rebind-services

xs login –o ORG –s SPACE2

xs rebind-services hana  or xs rebind-services

This will update the XS application environment variable if they are pointing to production

13.Reestablish connection to CICD tools like Jenkins by updating the passwords

14.Rebind the routes to correct space without downtime:

If post refresh xs routes | grep <oldspace name> entries are present, this must be corrected by following the below sequence to update the end points of the application

create-route -> map-route -> unbind-route -> delete-route

Note:If we unbind the current routes (spacex) before binding the new routes (space1), this will be a downtime activity . An application and a route are two different entities. Application is a process running on the XSA system, while a route that is binded to an application is an endpoint used to serve traffic for that application. If we need to update the routes without redeploying the application or without a downtime, the specified sequence must be followed. If we are fine with a downtime,then we can directly redeploy the mtas of that application where the routes gets updated automatically

Example:

When you execute xs routes | grep spack or xs apps | grep spack and you see entries like below

https://org-spack-sfsafas-xsjs-service.<host>:<port>

Here the entry *spack* is the space name of source system and hence we must have the URL of format below , follow the steps specified

https://org-space1-sfsafas-xsjs-service.<host>:<port>

To find the list of domains run xs domains.

NOTE:https://help.sap.com/viewer/1ed1948fa0664e138c088dcc61e267e0/2.0.04/en-US/7b24c9d9284643e49554e2eeeaad7be7.html

Steps:

a.Create a additional route as sidadm :

                xs create-route space1 <domain> -n <host name>

b.Map routes with the application

                As sidadm, xs map-route <app name> <domain> -n <hostname>

c.Unmap the old routes:

            As sidadm,

            xs unmap-route <App_name> <domain> -n <host>

d.Delete the unwanted route

As sidadm,

 xs delete-route <domain> -n <hostname>

e.Ensure that the applications are pointing to the correct route and space

xs app | grep space1 or xs app | grep space2

References:

https://help.sap.com/viewer/6b94445c94ae495c83a19646e7c3fd56/2.0.03/en-US/fe6459d09dfa4ba1b475cbd680df2474.html

https://launchpad.support.sap.com/#/notes/2300937