HCI: Deciphering HCI Keystore

Background

SAP provides 2 major operating models for HCI Installations,

• SAP Managed Operating Model – Major tasks for tenant related administration is handled by SAP including management of Keystore & Known_Hosts for SSH / SFTP Connections.
• Customer Managed Operating Model – Major Tasks for tenant administration is handled by the Customer. This model is currently applicable only for SAP HCI Partner Edition.

As most HCI Installations are on SAP Managed Operating Model –

• Have you ever wondered what the KeyStore from HCI contains?
• How does SSH / SFTP connectivity occur?

If yes, read on..

As-Is HCI Tenant Setup

As we start this journey to understand the setup, key important notes on the HCI Tenant:

• No Keystore is Deployed
• No Known_Hosts is deployed ( For SSH / SFTP Connections )

Learning#1 – Component CXF-endpoint-IFLMAP-hcibsp

Let’s start our journey of discovery with the  component CXF-endpoint-IFLMAP-hcibsp. Have you looked at the Components under your Tenant Management Node? Wondered what the Component: CXF-endpoint-IFLMAP-hcibsp does?

As per SAP Documentation, the role of this Component is to simulate an external SSL Call every 30 seconds to your HCI Tenant Run-time node from the Tenant Management Node via the SAP HCI Load Balancer. The status of this SSL call helps HCI confirm if the runtime node is up and running.

So what happens when you do not have a Keystore deployed on your HCI Tenant? Well, this Component goes into error with the message: “Cannot connect, no Keystore deployed” as SSL connections require a Keystore.

Create your Keystore – Use a Self-Signed Certificate

Well, let’s just go ahead and create our Keystore then, we thought. In our case it was a “Test” tenant and as we were keen to get our hands dirty on a “Fresh” HCI Tenant, we thought lets create a Self-Signed Keypair and deploy on the Keystore.

The steps to create the Keystore are described in this blog: Starting with hana cloud integration-keep this in mind . In our case we used a Self-Signed Certificate and deployed the same on the Keystore. This is what our Keystore looked like, i.e., just a single Self Signed Keypair,

Post deployment of the Keystore, the status of CXF-endpoint-IFLMAP-hcibsp changed to “javax.net.SSLPeerUnverifiedExcepton: peer not authenticated”

Learning#2 – Add HCI Tenant’s SSL Certificate to your Keystore

The SAP documentation clearly states a SSL call is made to the HCI Load Balancer which led us to believe that the SSL call was failing as the HCI LoadBalancer’s SSL Certificates were not trusted in the Keystore. This brought us to the next step, where we added our HCI LoadBalancers Certificate Chain to the Keystore.

Download BaseCertificate from your HCI URL

Download Intermediate Certificate from your HCI URL

Download Root Certificate from your HCI URL

Import all 3 Certificates into your Keystore. This is what our Keystore now looked like.

Deploy your Keystore with the hope that the error vanishes!

Learning#3 – Self-Signed Certificates – Sorry Not Accepted

Alas, the error continued!

We had the feeling at the back of our mind that Self Signed Certificates would probably not be accepted in the Keystore. SAP provides a list of CA’s whose Certificates are accepted by the Load Balancer for authentication which meant we had to now get our certificate signed by a Trusted CA as listed on SAP documentation here.

Update & Deploy Keystore with a Trusted CA

The Keystore was then updated and deployed with a Keypair signed by a TrustedCA.Subsequently the error disappeared, and the CXF-endpoint-IFLMAP-hcibsp turned into status Green 🙂

SSH / SFTP, Known_Hosts and Your Keystore

So, what role does the Keystore play when you need to deal with SSH/SFTP Connections? How do you generate a known_hosts file?

In the case of a SAP Managed Operating Model, these complexities are hidden to HCI Developers but what does SAP do behind the scenes?

Prerequisite:

The SFTP Server IP Address is open in the SAP Firewall. This action had to be taken by SAP and a ticket to SAP is the only way to go about the same.

Test your SSH Connection

Use the Test Outbound Connection  as described in the blog: HCI: Testing Outbound Connections from HCI to Test your connection to a SSH SFTP Server. The server returns the error “Retrieving known.hosts from cloud storage failed due to KeyStoreNotFoundException: Keystore with name: ‘known.hosts’”. The error clearly states – known_hosts is not deployed.

Learning#4 – Creating known_hosts file

A known_hosts file is a list of SFTP Servers and their public keys. This file enables your HCI Tenant know the list of Hosts that a SSH / SFTP Connection is allowed to.To generate a known_hosts file execute the below command from a SSH Terminal. In my case I use cygwin

Command: ssh-keyscan -t rsa <<IPAddress/HostName of SFTP Server>>

Copy the output of this command to a Text File and then deploy the same on the server.

SFTP Authentication & Keystore

So, now that our known_hosts is deployed let’s try a test connection we said. Alas, the error changed this time with error:  “Auth Fail”. Our SFTP Server had been set up for Key Based Authentication with the exact KeyPair that was loaded into our Keystore as described in the wiki: Generating SSH Keys for SFTP Adapters – Type 2

So why is our SFTP Server not authenticating us when the Key is available in the Keystore? How does HCI know which Keypair to be used for SFTP Authentication we wondered considering that there is no field to provide the Alias Name of your Private Key.

HCI does not provide you with an option to provide a Private Key Alias name for Key Based Authentication for SSH / SFTP. Instead HCI looks for Key’s with alias: id_rsa or id_dsa in the Keystore and uses this to authenticate itself.

Learning#5 – Update Keystore to have a entry for Keypair with id_rsa

To avoid any impact to existing scenarios, we copied the existing Keypair entry into a new entry called “id_rsa” and deployed the keystore.

Now when the SSH connection is tested, it works like a charm!

Conclusion for the HCI KeyStore

1. Ensure your Keypair is signed by a Trusted CA. ( No Self Signed Certificates )
2. Ensure your tenant’s Load Balancer Certificates are loaded into your Keystore.
3. Ensure a known_hosts file is created for SFTP Connections.Continue to append any new server to existing file.
4. For SFTP / SSH ensure your KeyPair has the alias: id_rsa or id_dsa