Technical Articles
Refreshing HANA systems involving XSA tenants
XSA specific HANA DB refresh steps:
In general, HANA refreshes are pretty straight forward and easy,.However, that is not the case when you have XS advanced tenants involved in your landscape. Many additional steps must be performed to bring back the target system to be fully operational with XSA applications without any issues.
This blog covers the pre-steps,refresh and post-steps specific to XSA associated HANA DB.
Note: This blog assumes that you are well aware of normal HANA refreshes and does not cover the normal HANA refresh related details. This documents assumes that the XSA is installed in separate tenant rather than on SystemDB .
Sample system:
Source | Target |
SystemDB | SystemDB |
QWE(XSA tenant :org ORG and space SPACEK) | DWE(XSA tenant :org ORG and space SPACE1) |
QER(XSA tenant :org ORG and space SPACEY) | DER(XSA tenant :org ORG and space SPACE2) |
QXS(XSA meta data tenant :org ORG and space SAP) | DXS(XSA meta data tenant :org ORG and space SAP) |
In this scenario we are going to refresh DWE with QWE,DER with QER and DXS with QXS . We will not be refreshing SystemDB as we are only considering the scenario where XSA is installed in tenant DB -QXS/DXS . This tenant QXS/DXS is called the meta data tenant of XSA
The same can be verified by logging into OS level using sidadm and executing command
XSA list-tenants
To explain above screen shot, XSA will have 1 org unit and multiple spaces.Each spaces will be associated with a tenant . Here I have represented org unit as ORG across the tenants and spaces as SPACE1 which is the space of tenant DWE and SPACE2 which is the space of tenant DER.
The tenant DXS is the tenant on which we have installed XSA and holds the meta data of the XSA persistence . This can be distinguished by the entry highlighted above which reads as “XS Advanced platform persistence : YES” and will be having the same org unit in a landscape (ORG) and SAP default space “SAP”
Prerequisites with regards to XSA tenant enabled system refresh:
1.Individual tenants must not be refreshed as a best practice . If a source system consists of 3,XSA tenants which includes 1 with XSA meta data, the target must also have same number of XSA tenants .
2.SAP recommends to perform as PIT refresh for all the XSA tenants in one SystemDB.
3.XSA will be disabled at the start of the refresh as a first step
4.You know the password for XSA_ADMIN which is a master user in XSA
A.Pre-refresh steps:
1.Identify the XSA cockpit URL
Login to XSA meta data tenant and inside SAP space ,identify the XSA cockpit URL if not knownAs sidadm,
xs login –o ORG -s SAP
Once inside the SAP space, type
xs apps
From the above output identify the XSA-cockpit URL.
2.Open the URL in browser and take the pre-snaps of important data
A.Spaces:
Click on each spaces and record the screen shot of applications,services,User provided Services,Routes etcWe can get these details even from OS level by logging into each space and collect information
-To login to each space xs login-o <ORG> -s <SPACE1>
– To capture all the application running in the space -> xs apps
-To capture the services in each space -> xs services
-To capture the routes in each space -> xs routes
-To capture the list of MTARs deployed on each space -> xs mats
B.Users
In our case post refresh we will have users only from source and hence we record the list of users in XSA to be recreated post refresh.
-To find the list of XSA users, goto XSA cockpit and click on user management
C.Users and space mapping:
Each of the users above will have access to individual space only if there is an explicit mapping of Space roles like SPACE DEVELOPER,SPACE MANAGER etc to the users in the respective spaces.
Click on each space from XSA cockpit home page and go to members and capture the screen shot for each space members, so that same can be validated and recreated post refresh
D.Current custom role collections.
Goto cockpit home page and click on Role Collections .
If there are some custom role collections that is not present in source, that screen shot can be captured . Click on each role collections to capture the screen shot of associated roles inside each role collections.
E. Exporting trust configuration:
If you have AD connected to the XSA for SSO , then we can download the meta data of original system and can be reimported again post refresh
F.Take screenshots of configuration file:
From System DB take a screen shot of below in case of SSO configuration in place
G.Take a note of api URL in target
As sidadm execute the command xs api
<sid>adm@sddgfdgg:/usr/sap/<SID>/SYS/global/hdb/custom>xs api
API endpoint: https://<host>:<port>
SSL trust: The authenticity of host ‘xx.cdscdsf.com’ is established by the ‘/hana/shared/<SID>/xs/controller_data/controller/ssl-pub/router/default.root.crt.pem’ certificate.
H.Copy entries of UPS (User provided services)
Post refresh, as all the UPS must be brought back with original configuration as it will hold production data post refresh. Hence click on UPS entries and take screen shot of each of the entry
I. Make a note of general health check on XSA
As sidadm , execute XSA diagnose command
J.Ensure source and target with respect to xsa_sizing parameter is same to avoid memory issue or XSA instability issues post refresh
K.Login to each space and take screen shot and make a note of API url with below command:
xs login –o ORG –s SPACE1
xs login –o ORG –s SPACE2
xs login –o ORG –s SAP
B. Refresh:
1.Before starting with refresh disable the XSA services
As sidadm, execute XSA disable
After this all the xsa related services will be down.
2.Take a backup of the File-System Service (FSS) in an XS advanced backup operation, run the backup-fss command as <sid>adm,
XSA backup-fss
3.Perform a PIT restore from source to target w.r.t to the decided PIT on all the target tenants respectively using normal HANA restore method.
QWE(XSA tenant :org ORG and space SPACEK) |
DWE(XSA tenant :org ORG and space SPACE1) |
QER(XSA tenant :org ORG and space SPACEY) |
DER(XSA tenant :org ORG and space SPACE2) |
QXS(XSA meta data tenant :org ORG and space SAP) |
DXS(XSA meta data tenant :org ORG and space SAP) |
QWE will be restored with the hh:mm:ss to DWE
QER will be restored with the hh:mm:ss to DER
QXS will be restored with the hh:mm:ss to DXS
Post successful completion, your XSA related services will still be disabled.
4.After completing the restore successfully , disable the XSA services if found active.
Execute command XSA disable as sidadm
5.Now point the XS advanced platform services to the tenant database containing the XS advanced platform data we just restored, for example, by running the following command as <sid>adm user, execute below command
XSA select-xsa-runtime-db DXS
This command also restores the fss backup that we had took in step 2, XSA backup-fss along with enabling the XSA services that were red w.r.t screen shot shown in step 1.All services in systemdb will be green by now.
6.Run XSA health check and make a note of all the errors if any .Ensure all the XSA tenants are intact . Update the tenant password if needed.
XSA diagnose
7.Rename the XSA tenant space to original target
After refresh, we see that the spaces that are connected to the target is similar to source system and hence we need to rename the space.
You can check the same by executing
XSA list-tenants
Now by referring to earlier details of source and target spaces, rename the target spaces which will be using the source space name , to original target
As sidadm, execute the below command
xs rename-space SPACEK SPACE1
xs rename-space SPACEY SPACE2
Now rerun XSA list-tenants to check the tenants, and their space mapping .
To display the spaces alone you can uses command xs spaces
8.Reapply SSL certificate as the XSA cockpit URL will not show insecure if it was secured earlier.
As sidadm execute below. (Refer last prestep f for the URL)
xs l -a https://<fdsf.dfsdfsf.dgsgfs.net>:nnnnn–skip-ssl-validation
then,
cccadm@sfsfasfsaf:/hana/shared/CCC/HDB00/sapghercsw/sec>openssl pkcs12 -in star.dfsdfggfgg net.p12 -out star.sdasfsdfgdsg.pem -nodes
Enter Import Password:
MAC verified OK
Now,Set the certificate
xs set-certificate <URL.net> -k pkey -c chain
Now restart XSA
As sidadm, execute the command -> XSA restart
9.Login to each space and perform a general check if any
xs login –o ORG –s SPACE1
xs login –o ORG –s SPACE2
xs login –o ORG –s SAP
10.Now login to XSA with the API endpoint URL that we get from previous step
As sidadm
xs l -a https://>fdsf.dfsdfsf.dgsgfs.net>:nnnnn–skip-ssl-validation
11.Based on the screen shots taken during prerefresh update below in xsa cockpit URL
-UPS (User provided services)
-Users
-Space user mapping
-Role and role collection mapping
-Reimport the meta data and ensure that the entity id matches based on pre-step e =>login.entityid
12.Rebind all the xs application in each space using the command below .This will ensure that all the production environment variable in applications will get updated
Login to each space and execute below command
xs login –o ORG –s SPACE1
xs rebind-services hana or xs rebind-services
xs login –o ORG –s SPACE2
xs rebind-services hana or xs rebind-services
This will update the XS application environment variable if they are pointing to production
13.Reestablish connection to CICD tools like Jenkins by updating the passwords
14.Rebind the routes to correct space without downtime:
If post refresh xs routes | grep <oldspace name> entries are present, this must be corrected by following the below sequence to update the end points of the application
create-route -> map-route -> unbind-route -> delete-route
Note:If we unbind the current routes (spacex) before binding the new routes (space1), this will be a downtime activity . An application and a route are two different entities. Application is a process running on the XSA system, while a route that is binded to an application is an endpoint used to serve traffic for that application. If we need to update the routes without redeploying the application or without a downtime, the specified sequence must be followed. If we are fine with a downtime,then we can directly redeploy the mtas of that application where the routes gets updated automatically
Example:
When you execute xs routes | grep spack or xs apps | grep spack and you see entries like below
https://org-spack-sfsafas-xsjs-service.<host>:<port>
Here the entry *spack* is the space name of source system and hence we must have the URL of format below , follow the steps specified
https://org-space1-sfsafas-xsjs-service.<host>:<port>
To find the list of domains run xs domains.
NOTE:https://help.sap.com/viewer/1ed1948fa0664e138c088dcc61e267e0/2.0.04/en-US/7b24c9d9284643e49554e2eeeaad7be7.html
Steps:
a.Create a additional route as sidadm :
xs create-route space1 <domain> -n <host name>
b.Map routes with the application
As sidadm, xs map-route <app name> <domain> -n <hostname>
c.Unmap the old routes:
As sidadm,
xs unmap-route <App_name> <domain> -n <host>
d.Delete the unwanted route
As sidadm,
xs delete-route <domain> -n <hostname>
e.Ensure that the applications are pointing to the correct route and space
xs app | grep space1 or xs app | grep space2
References:
https://launchpad.support.sap.com/#/notes/2300937
Thanks for detailed steps.
One key info is that XSA backup should be taken along with db backup on source or just few minutes other wise data inconsistency issue will arise while recovery.
My 2 cents from my experience. Plzz update blog.
Hi RK, Thanks for your inputs. However, this blog was purely written based on my refresh experience.
Please note that it is a mandatory prerequisite to perform a PIT restore of all the tenants involved including meta data tenant as specified in this blog
Hello Rajeswari,
Thank you for sharing steps.
here i am doing the sandbox refresh from production,but i want to keep the old xs ,so i took backup using the command "XSA backup-fss" and xs version is 1.0.132.593
after the restore of SYSTEMDB and TENANT DB PIT ,i restored the backup that i initially took using the command "XSA select-xsa-runtime-db SYSTEMDB"
then execute the XSA diagnose i dont see any issues in output
when i execute the "xs rebind-services hana" to bind the services.
i am seeing below error, Please suggest
FAILED: Asynchronous job 'Creating service binding between app 'di-core' [Org 'ABCD', Space 'SAP'] and service instance 'di-core-hdi' [service: 'hana', plan:'hdi-shared'] of [Org 'ABCD', Space 'SAP'].' failed: Received error from service broker: REST request PUT to url 'https://HOSTNAME:30032/hdi-broker/v2/service_instances/5029ec20-80d8-4473-a936-f94a88fc6506/service_bindings/43c4b519-4b6b-4735-87ac-889420ed697e' had response code 500 (Operation failed due to unexpected server error) with error message: 'description: Failed to bind container '2EC0640D7B8B4CA1A73AB9E21D859E82' on database '10aef8c4-3074-4e4e-b602-3fdf648f2bd5' (HOSTNAME:30013), because of: Container '2EC0640D7B8B4CA1A73AB9E21D859E82' does not exist on the database'..
Is you XSA tenant SystemDB? This document is only specific to scenario where your tenant db is the XSA meta data . Based on the runtime command that you have pasted, it looks like you have installed your xsa on systemdb ?
Also diserver must be present in the system. If not please add it manually by using "ALTER SYSTEM RECONFIGURE SERVICE (<service_name>,<hostname>,<port_number>)"
Hello Rajeswari,
diserver is already running on SYSTEMDB.
Thanks
Now i am getting below error
FAILED: Asynchronous job 'Starting 1 instance(s) of app 'di-core' [Org 'ORG', Space 'SAP']' failed: com.sap.xs2rt.controller.ControllerExceptions$StartApplicationException: No Execution Agent available to start app 'di-core' [Org 'ORG', Space 'SAP'].
Please check if you have diservice agent in services. If not please add it .
Hi @Rajarajeswari Kaliyaperuumal. Do you have some steps related to XSA SSL enablement with w.r.t to host based routing . i have have below error on getting SSL signed cert import
I have followed below S-Notes in performing
2487639 - HANA Basic How-To Series - HANA and SSL - LEAD KBA
2243019 - Providing SSL certificates for domains defined in SAP HANA extended application services, advanced model
Hello Rajajrajeshwari,
I need to migrate On-Premise HANA DB to Cloud HANA DB having XSA. So, how do we handle this scenario?
Regards,
Amit Bharti