XSA/Cockpit installation Issues and Troubleshooting

Overview: HANA 2.0 is having XSA as base for many application like HANA cockpit, WEB IDE, Big Data Hub to name a few. You are lucky if you get it through for installation but sometimes you are stuck with few issues which you keep trying to solve , you don find solution. This Blog identifies and help you solve few of them.

Getting a guide is very easy. But troubleshooting in real world might make your head spin. So sharing my experience.

XX will represent the instance no. used in this blog.

Error:

1.  Adding role ‘xs_worker’ failed:

Check /var/tmp latest update folder and it will mention the file name which contains more details. It should be with name like hdbmodify_addlocalroles.log. and

Reason:  FAIL: process hdbxscontroller HDB XS Controller not running

Sub Reason:  Could not startup the controller components: SAP WebDispatcher failed to bind application port (already occupied). Route to application will be unavailable. Restarting the application may solve the issue.

xscontroller.out: FAILED to start: Could not startup the controller components: SAP WebDispatcher failed to bind application port (al
ready occupied). Route to application will be unavailable. Restarting the application may solve the issue.

Solution: Clearly xscontroller is not up and hence webdispatcher can’t bind.

1. Try to find out the port using netstat -a|grep 3XX29 where xx is the instance no. If you find the port being used by some random process, you can kill the process , try to start XScontroller again (using HANA studio or CLI) and proceed
2. sometime you don find any port used , in that case ,you can restart the entire application and try to start xscontroller again.
3. if 2nd does not work, you can try to kill all processes after shutting down HANA system, and then try to restart xscontroller using HDB start. sometime sapcontrol -d -nr XX -function StartSystem.

2. com.sap.xs2rt.installation.util.InstallationException: Error during serviceBrokerClient.getServiceBrokers()  UAA’s check_token() endpoint was called with invalid credentials. Could be a clientsecret mismatch

Reason: Such issue can occur when previous installation is not successful or credentials are mismatched

Solution: Remove previous installation/upgrade of XSA. using below steps:

1. Remove xs roles using hdblcm from installation directory 9/hana/shared/<SID>/hdblcm) which can be different as per customer.
2. uninstall XS component using hdblcm from same directory, but only xsa component, not entire DB or HDB client or something.
3. Restart the installation/upgrade. it will work

3. /installation/update-content: Cannot upload the data.

Reason: There can be 2 reasons

1. Network might be intermittently unavailable
2. zip or other extract utility missing from store. (zypper or yum or other depending on your OS)
3. Authorization issue/ownership issue.

Solution

1. Make sure network is available , no DNS/Hostname etc issue should cause installation/upgrade to stop.
2. install the zip or extract utility used by installer.
3. make sure authorization and ownership do not cause the halt of installation.

4. Update not starting : unhandled exception: Can’t call method “getManifest” on unblessed reference at LCM/Utils/ComponentActionStringGenerator/InstallUpdate.pm line 109.

Reason: There are some files pending from where the ./hdblcm is getting information about previous attempt of update/installation.

Solution: Usually there will be xml available in installation directory where hdblcm resides.

files might be like hdblcm.update or hdm.pid . It does not matter where are you running it from. But you will find such flles always in installation directory

Will keep posting more in case faced some issues further , of course with resolutions.

Troubleshooting Tips:

XSA is quite tricky and when it comes to troubleshooting , there is not one place you can find the logs:

1. cdtrace->/usr/sap/<SID>/HDB<nr>/<hostname>/trace and ls -lart , XS related stuff can be found in xsexeagent_0.log, xsuaaserver.out,uaa.log,xscontroller.out,hdbxscontroller_audit.log etc.
2. /hana/shared/<SID>/xs under this directory (/hana/shared/<SID>/xs/controller_data/controller/router/webdispatcher). routing details can be found along with logs.
3. if you are updating/installing you can find more details in /var/tmp and latest files. That is hopefully known to all.

5. HANA 401 unauthorized access

:HDB:dh1adm /usr/sap/DH1/HDB02 51> xs uninstall <Product_Name> N-PREMISE
Really uninstall ‘Product_Name’ ? (y/n)> y

Running uninstallation of “Product_Name” in org “orgname” / space “SAP” as xsa_admin…
ERR javax.ws.rs.NotAuthorizedException: HTTP 401 Unauthorized

Reason: We have to explicitly login using xs login

Solution: Xs login and provide credentials. CLI is faster than UI XSA.

Suggestion: If UI part of XSA lifecycle is having time out issues and taking time for deployment/deregistration etc , try using XS CLI, it is way too faster than UI as of now.

6 .Could not retrieve the download URL for Node.js: Error: certificate has expired

Error: Could not retrieve the download URL for Node.js: Error: certificate has expired (connecting to IP:Port)

Execution of bin/compile failed (exit value 1)
09:14:50.997 – INFO: Output line 111: [28] 5/18/18 9:14:50.792 AM [STG/4] ERR Failed to stage app ‘auditlog-db’ [Org ‘HANACockpit’, Space ‘SAP’]: Execution of bin/compile failed (exit value 1).

Reason: Certificate issue

Solution:  Reset Certificate using command XSA reset-certificate

XSA reset-certificate
Stopping XSA …
Starting XSA …
OK

you can provide customer provided certificates directly.

6. mtaext files causing Cockpit installation to fail:

Error:  HANACOCKPIT.mtaext file is wrong and installation cannot proceed. Invalid inputs.

Reason: The File is changed . The task for passing this file is to fix the ports for cockpit web and web app. Even an extra/less space may cause disruption.

Solution: Download the exact file from sap Note and change only port no. nothing else.

2389709 – Specifying the port for SAP HANA Cockpit before installation

7. HANA cockpit or XSA application performance issue on Hyperscalers AWS, Azure GCP 

Error: XSA applications shows significant performance impact on public cloud platform

Reason: Seems issues with performance of XSA on hyperscalers. ‘XSA diagnose’ tool can help us in this case.

login with xs admin and run XSA diagnose

Checking operating system user setup (space users): Unexpected error, see log file: FAILED

File system: Checking realpath performance: OK

HANA: Checking INI parameter configuration: OK

File system: Checking write performance: FAILED

Thread count   Dir. Count   File Count   Package Size(MB)   Throughput(MB/s)   Write Performance

————————————————————————————————

4              4000         24000        398.44             6.36               SLOW

————————————————————————————————

The file system is too slow for XSA to work properly.

Solution: Mount block devices for xs directory and install xs on it. Then performance looks fine.