SAP Cloud Connectivity issues due to Expired Certificate
A number of customers over the last few months experienced a sudden loss of connectivity in their SAP Cloud Platform landscapes. The customer’s environments had already been live for sometime so when receiving a call about them totally losing connectivity was odd. I had not heard of this before even though being experienced in this area. So, with a bit of digging – and of course experiencing it myself I thought to share with the SAP Community. With Productive landscapes now being in operation for a few years I would imagine a lot of customers would start to experience this so hopefully this blog post can help and allow them to be proactive.
***Update 14/01/2020 – New version 2.12.2 shows the subaccount certificate validity when getting close to the end date.” Further details below.
What is the Cause?
The sudden loss of connectivity is due to certificates expiring on SAP Cloud Platform – specifically for the subaccount. Each subaccount (I found out) has a number of certificates, and they can expire! A simple action to renew the certificate must be performed – but it must be performed from the SAP Cloud Connector application. To me, this sort of thing should just be automated however at this stage I don’t believe there is any automated functionality to do this. There is functionality to set up email but most of the clients I have worked have still not set this up. Definitely a #FeatureRequestPhil item!
When this occurs you will see a message appear in the Cloud Connector administration summary page. You will see the Disconnected icon as well as the message “Invalid status of handshake response: 401 Unauthorized”.
Figure:1 Invalid status – Unauthorised message
If you check the Alerting page you will also see the detailed messages telling you that the certificate has in fact expired. These ones are more detailed and does tell you what has happened. It does tell you that the tunnel connection is broken and cannot be used. This means that applications will not run at all in that particular subaccount.
Figure:2 Detailed Alerting messages – Certificate expired
On the subaccount dashboard you will also see that in the list of subaccounts that the Status is red.
Figure:3 Subaccount Dashboard – Status is broken due to certificate expiry
So, how can you check the validity dates of the certificates in SAP Cloud Platform so that you can be proactive? It is actually quite simple – you need to use console commands to interrogate the keystore, specifically checking the validity on certificates held within the keystore. Before running the commands I will provide some background to assist with the understanding.
To understand the keystores I would recommend you logging into the server where the Cloud Connector resides and using a File Explorer to check the following location for the keystore file path. Here is an example for my trial account but the same applies to any live systems.
Figure:4 Certificate keystore file path for SAP Cloud Platform via SAP Cloud Connector
You can find the keystore using the following path. I have assumed this is located on the C:drive.
c:\SAP\scc20\scc_confg\host name\sub account technical name\scc.jks
The key store file we are checking is scc.jks.
A real example of the above can be viewed in the below screenshots and if you have multiple subaccounts linked to the SAP Cloud Connector you will see a number of subaccount technical name folders in the File Explorer. Each subaccount will have it’s own keystore (scc.jks file) so be sure to check each for certificate expiry dates.
Figure:5 Subaccount folders within the SAP Cloud Connector
A specific keytool console command is required and I’ve found that the easiest way to carry this out is to navigate to the directory that contains the keytool from the SAP JVM. This will just ensure it runs successfully. In my experience I always locate the SAP JVM on the C: drive of the server itself and usually install the SAP Cloud Connector on a separate drive (e.g. D: drive). This means when you run the command you need to reference the location of the SCC installation. The screenshot below shows this.
To run the command you need to use the Command prompt. The full command is as follows.
COMMAND = keytool -list -v -keystore D:\SAP\scc20\scc_config\Host Name\Neo subaccount technical name\scc.jks
Note: I have used the D: drive above but the drive name needs to be the location where the SAP Cloud Connector application is installed. This is a Windows installation example.
The following information is referenced in the above keytool command including:
- Host name e.g. ap1.hana.ondemand.com
- Neo Sub-Account technical name.
You will then need to run the specific command to list the certificates in the keystore.
keytool -list -v -keystore D:\SAP\scc20\scc_config\ap1.hana.ondemand.com\c34dxxxxx\scc.jks
When you run the command you will need to enter the keystore password as you can see below. At first I had no idea what this would be. I tried my S user password for SAP Cloud Platform and also tried the Cloud Connector S userid password but none of them worked. I had not previously set any password for the keystore so just assumed there was none and this worked. I pressed [Enter] to proceed without a password and this was ok.
So, for the keystore password (unless you have set one) just press [Enter] to proceed.
The actual example I ran recently is included below.
Figure:6 Results from the keytool keystore command to check the Certificate keystore
As you can see from above, the keystore contains 2 entries. The first one is the CA certificate detailed by the Alias name = ca. This was created on the 24/01/2019. If you check the Validity period you can see this is valid until Mon April 22 2024 – so the good news is we still have a little to go before we need to renew this certificate.
The next one is the actual subaccount certificate. This is the one we are interested in renewing.
The alias name in this case will equal the subaccount technical name.
Figure:7 Subaccount certificate expiry information from the keystore
This was also created on the 24 Jan 2019. As shown above you can see the validity period of the certificate and it expires soon. The certificate is valid until Jan 24th 2020 so it will need to be renewed soon to avoid any connectivity issues.
With this new found information we can now construct a list of each subaccount with the dates that the certificates will expire. This should be added to normal BAU (Business As Usual) maintenance activities in organisations.
Renewing the Certificate
So, now that we know the validity dates we can now plan to renew them. This is carried out in the SAP Cloud Connector. In the Cloud Connector administration page you will see the [Renew Subaccount Certificate] icon up in the top right hand corner.
Figure:8 Subaccount certificate renewal button in SAP Cloud Connector
To renew the certificate click on the [Renew Subaccount Certificate] button. A popup window will be displayed asking for the username and password.
Figure:9 Subaccount credentials pop-up screen
Enter your SAP Cloud Platform username and password. This is your S or P userid for access SAP Cloud Platform global accounts or subaccounts. Click on [OK] to confirm and you will see a “Changes were saved” message.
You will need to reconnect to the subaccount because connectivity was lost due to the certificate expiring. To do this simply click on the [Connect] button as displayed below.
Figure:10 Re-Connect to the Subaccount screen
Once this is carried out the subaccount will come back online and that previous message will disappear. You should now see a positive connector state with a Status of “Connected” shown as displayed below.
Figure:11 Connected screen showing “Connected” status
If you now look at the [Alerting] section you will see information about the certificate renewal and the subsequent validity end date. Make sure you note this down!!! 🙂
Figure:12 SAP Cloud Connector Alerting screen
As you can see I had expired certificates in multiple subaccounts so had to renew a number of them in December….hence I worked out pretty quickly how to fix them. 🙂
Crisis averted – with following these activities we should not experience any downtime but definitely suggest keeping a record of all of the expiry dates of certificates from all subaccounts.
Version 2.12.2 – Upgrade
Since posting this I have found out through Markus Tolksdorf that version 2.12.2 provides more information on the Subaccount certificate. Specifically displaying whether it is valid or not. So, I thought I would provide some screenshots on exactly what this new version displays.
I upgraded my local version of the Cloud Connector and really great to see the additional fields – check below. To do this you can follow previous blogs I have written. For Windows follow this guide here or for Linux follow this guide here.
Figure:13 Version 2.12.2 Cloud Connector showing Subaccount Certificate field
As you can see above there is a new field called Subaccount Certificate – shown as 1 above. Funnily enough mine had expired. You will see that the same error message is displayed about Invalid status of Handshake. Perfect timing :-). As detailed above, we need to click on the [Refresh subaccount certificate] button shown as 2 above. This will display a pop-up window requesting your User name and password. As also detailed above you will need to enter your SAP Cloud Platform username and password.
Figure:14 Refresh Certificate popup screen
Once this is done the Certificate will be automatically renewed.
Figure:15 Certificate Valid screen
You will also see that the Certificate is now valid. If you check the Alerting section you will also see that the certificate was renewed and you will also see the new Valid to date.
Figure:16 Alerting screen showing new Certificate valid to date
As the connection dropped due to the Subaccount certificate expiring you will also need to re-connect. This was already covered above.
I strongly recommend upgrading your version of the SAP Cloud Connector. By default there will be a date shown of the certificate expiry when is getting close to the renewal date (within 30 days) so definitely worth performing the upgrade.
Lastly, I am definitely no expert in this area and still need to work out what happens with the CA certificate renewal but will provide an update in this blog once I find out. Of course if anyone can supply how this is renewed that would be helpful to add.
As always, thanks for reading!!!