Skip to Content
Author's profile photo Bhavesh Kantilal

HCI: Deciphering HCI Keystore

Background

SAP provides 2 major operating models for HCI Installations,

  • SAP Managed Operating Model – Major tasks for tenant related administration is handled by SAP including management of Keystore & Known_Hosts for SSH / SFTP Connections.
  • Customer Managed Operating Model – Major Tasks for tenant administration is handled by the Customer. This model is currently applicable only for SAP HCI Partner Edition.

As most HCI Installations are on SAP Managed Operating Model –

  • Have you ever wondered what the KeyStore from HCI contains?
  • How does SSH / SFTP connectivity occur?

If yes, read on..


As-Is HCI Tenant Setup

As we start this journey to understand the setup, key important notes on the HCI Tenant:

  • No Keystore is Deployed
  • No Known_Hosts is deployed ( For SSH / SFTP Connections )

Learning#1 – Component CXF-endpoint-IFLMAP-hcibsp

Let’s start our journey of discovery with the  component CXF-endpoint-IFLMAP-hcibsp. Have you looked at the Components under your Tenant Management Node? Wondered what the Component: CXF-endpoint-IFLMAP-hcibsp does?

As per SAP Documentation, the role of this Component is to simulate an external SSL Call every 30 seconds to your HCI Tenant Run-time node from the Tenant Management Node via the SAP HCI Load Balancer. The status of this SSL call helps HCI confirm if the runtime node is up and running.

So what happens when you do not have a Keystore deployed on your HCI Tenant? Well, this Component goes into error with the message: “Cannot connect, no Keystore deployed” as SSL connections require a Keystore.

/wp-content/uploads/2016/03/1_919224.png

Create your Keystore – Use a Self-Signed Certificate

Well, let’s just go ahead and create our Keystore then, we thought. In our case it was a “Test” tenant and as we were keen to get our hands dirty on a “Fresh” HCI Tenant, we thought lets create a Self-Signed Keypair and deploy on the Keystore.

The steps to create the Keystore are described in this blog: Starting with hana cloud integration-keep this in mind . In our case we used a Self-Signed Certificate and deployed the same on the Keystore. This is what our Keystore looked like, i.e., just a single Self Signed Keypair,

/wp-content/uploads/2016/03/2_919274.png

Post deployment of the Keystore, the status of CXF-endpoint-IFLMAP-hcibsp changed to “javax.net.SSLPeerUnverifiedExcepton: peer not authenticated”

Learning#2 – Add HCI Tenant’s SSL Certificate to your Keystore

The SAP documentation clearly states a SSL call is made to the HCI Load Balancer which led us to believe that the SSL call was failing as the HCI LoadBalancer’s SSL Certificates were not trusted in the Keystore. This brought us to the next step, where we added our HCI LoadBalancers Certificate Chain to the Keystore.

Download BaseCertificate from your HCI URL

/wp-content/uploads/2016/03/3_919275.png

/wp-content/uploads/2016/03/4_919276.png

Download Intermediate Certificate from your HCI URL

/wp-content/uploads/2016/03/5_919289.png

Download Root Certificate from your HCI URL

/wp-content/uploads/2016/03/6_919290.png

Import all 3 Certificates into your Keystore. This is what our Keystore now looked like.

/wp-content/uploads/2016/03/7_919291.png


Deploy your Keystore with the hope that the error vanishes!


Learning#3 – Self-Signed Certificates – Sorry Not Accepted

Alas, the error continued!

We had the feeling at the back of our mind that Self Signed Certificates would probably not be accepted in the Keystore. SAP provides a list of CA’s whose Certificates are accepted by the Load Balancer for authentication which meant we had to now get our certificate signed by a Trusted CA as listed on SAP documentation here.


Update & Deploy Keystore with a Trusted CA

The Keystore was then updated and deployed with a Keypair signed by a TrustedCA.Subsequently the error disappeared, and the CXF-endpoint-IFLMAP-hcibsp turned into status Green 🙂

/wp-content/uploads/2016/03/8_919310.png



SSH / SFTP, Known_Hosts and Your Keystore

So, what role does the Keystore play when you need to deal with SSH/SFTP Connections? How do you generate a known_hosts file?

In the case of a SAP Managed Operating Model, these complexities are hidden to HCI Developers but what does SAP do behind the scenes?


Prerequisite:

The SFTP Server IP Address is open in the SAP Firewall. This action had to be taken by SAP and a ticket to SAP is the only way to go about the same.

Test your SSH Connection

Use the Test Outbound Connection  as described in the blog: HCI: Testing Outbound Connections from HCI to Test your connection to a SSH SFTP Server. The server returns the error “Retrieving known.hosts from cloud storage failed due to KeyStoreNotFoundException: Keystore with name: ‘known.hosts’”. The error clearly states – known_hosts is not deployed.

Learning#4 – Creating known_hosts file

A known_hosts file is a list of SFTP Servers and their public keys. This file enables your HCI Tenant know the list of Hosts that a SSH / SFTP Connection is allowed to.To generate a known_hosts file execute the below command from a SSH Terminal. In my case I use cygwin

Command: ssh-keyscan -t rsa <<IPAddress/HostName of SFTP Server>>


/wp-content/uploads/2016/03/9_919312.png

Copy the output of this command to a Text File and then deploy the same on the server.

/wp-content/uploads/2016/03/10_919328.png

/wp-content/uploads/2016/03/11_919329.png

/wp-content/uploads/2016/03/12_919330.png

/wp-content/uploads/2016/03/13_919331.png

SFTP Authentication & Keystore

So, now that our known_hosts is deployed let’s try a test connection we said. Alas, the error changed this time with error:  “Auth Fail”. Our SFTP Server had been set up for Key Based Authentication with the exact KeyPair that was loaded into our Keystore as described in the wiki: Generating SSH Keys for SFTP Adapters – Type 2

So why is our SFTP Server not authenticating us when the Key is available in the Keystore? How does HCI know which Keypair to be used for SFTP Authentication we wondered considering that there is no field to provide the Alias Name of your Private Key.

HCI does not provide you with an option to provide a Private Key Alias name for Key Based Authentication for SSH / SFTP. Instead HCI looks for Key’s with alias: id_rsa or id_dsa in the Keystore and uses this to authenticate itself.

Learning#5 – Update Keystore to have a entry for Keypair with id_rsa

To avoid any impact to existing scenarios, we copied the existing Keypair entry into a new entry called “id_rsa” and deployed the keystore.

/wp-content/uploads/2016/03/14_919332.png

Now when the SSH connection is tested, it works like a charm!

/wp-content/uploads/2016/03/15_919333.png



Conclusion for the HCI KeyStore

  1. Ensure your Keypair is signed by a Trusted CA. ( No Self Signed Certificates )
  2. Ensure your tenant’s Load Balancer Certificates are loaded into your Keystore.
  3. Ensure a known_hosts file is created for SFTP Connections.Continue to append any new server to existing file.
  4. For SFTP / SSH ensure your KeyPair has the alias: id_rsa or id_dsa

Assigned Tags

      21 Comments
      You must be Logged on to comment or reply to a post.
      Author's profile photo Former Member
      Former Member

      Hi Bhavesh,

      Very nice blog and informative. Thanks for sharing.

      One thing to confirm since you brought up the Operating Model for HCI Installation. I know that first one (managed by SAP), the second one is not that I am aware. Does this mean that there's an on-premise installation for SAP HCI and is released already to customers and available for vendors? Or this is related to Netweaver 7.5 (SAP PO) which has the functionality on using the integration content of SAP HCI?

      Cheers,

      R-jay

      Author's profile photo Bhavesh Kantilal
      Bhavesh Kantilal
      Blog Post Author

      Hello Ray,

      This has nothing to do with an OnPremise SAP PO 7.5 and the ability to deploy Cloud Integration Content.There is no on-premise Installation of HCI.


      The HCI Installation continues to be on the Cloud. Its like any normal HCI Instance on the Cloud. The Customer Managed Operating Model is available in the in HCI Instances under the license model  "SAP HCI Partner Edition".

      As this is a Partner Instance of HCI on SAP Cloud, SAP lets all Administration activities like deployment of keystores to be done by the Partner itself unlike the SAP Managed Model.

      You may read more on this in link and its subsections: SAP HANA Cloud Integration

      Regards,

      Bhavesh

      Author's profile photo Former Member
      Former Member

      Hi Bhavesh,

      Clear for me now. Thank you!

      Cheers,

      R-jay

      Author's profile photo Harish Mistri
      Harish Mistri

      Thanks Bhavesh for sharing !!!! this will solve lot of issue and doubts around secure connections !!!

      Author's profile photo Bhargav Gogineni
      Bhargav Gogineni

      Thanks for Information Bhavesh. Very Informative!

      Bhargav

      Author's profile photo Former Member
      Former Member

      Hello, Thank you for that nice and helpfull blog post. I have a question about how to sign a certificate with a trustet CA like goDaddy.

      I uploadet my CSR to goDaddys webpage but now I am stuck because I have to do the steps desribed under the following link:http://help.secureserver.net/article/7452?locale=en

      Thanks a lot for your answer.

      Author's profile photo Bhavesh Kantilal
      Bhavesh Kantilal
      Blog Post Author

      Hello Stefan,

      we had the same issue with GoDaddy. Instead we took one of our existing key pair on another server and loaded the same in this key store as this was just used to make sure that the connection for internal HCI load balancer works.

      if you are in a hurry to get your HCI tenant operational one tip-

      • If you have a service market Place ID get your SSO certificate from service market place ( install in browser)
      • EXport this private key from your browser and then load this as your key pair in your HCI key store.
      • This key is signed by SAP CA public certificate and hence also make sure SAP CA from service market place is loaded into your key store.

      Regards

      Bhavesh

      Author's profile photo Former Member
      Former Member

      Hello Bhavesh

      Thanks a lot for your answer. I tried it and it works. At the Moment I stuck with the next failure message but I still work on it. 😉

      At the Moment I really don't Know How to Setting up our Webdispatcher and the HANA Cloud Connector. If I do the Connection test in C4C I'll get the following message in the HCI:

      Inbound processing in endpoint at /COD/ERP/SimpleConnect failed with message "Fault:Could not send Message.", caused by "SunCertPathBuilderException:unable to find valid certification path to requested target"

      But do you Know How does it works if you really want to sign a certificate? Is there any possibility to sign the certificate by SAP with a OSS Call or something like that?

      A lot of questions, thank you for your further help.

      Regards, Stefan

      Author's profile photo Bhavesh Kantilal
      Bhavesh Kantilal
      Blog Post Author

      Hello Stefan,

      Is your Tenant under a Partner Managed Model or SAP Managed Model? If SAP Managed Model, the keystore is controlled by SAP and SAP will generate the Keys for you. If partner managed, in my case we just used an existing key pair and left it at that as it has no impact to the other scenarios.

      From the error it looks like, the message has arrived at HCI from C4C. When HCI tries to deliver this message to your ERP tentant the same fails with a SSL error.

      This can be a extensive topic and hence would suggest you raise a question on the PI forum and I would be happy to assist there but prior to that couple of checks,

      • If you are using a direct connection from HCI to ECC without HCC, then make sure your SSL certificate is loaded into the HCI keystore.
      • If you are using HANA Cloud Connector, has HCC been able to connect to your ECC backend?

      Regards,

      Bhavesh

      Author's profile photo Horst Müller
      Horst Müller

      Hello Bhavesh,

      our HCI is running under "customer managed operational model" and I'm tring to set up a connection ERP - HCI - C4C (and back).

      I've created a keystore, created a CSR and received a certificate (using Keystore Explorer)

      Could you explain where and how the keystore could be imported. I can't find the component "CXF-endpont-IFLMAP-hcibsp" you described.

      Thanks for your Response!

      Horst

      Author's profile photo Bhavesh Kantilal
      Bhavesh Kantilal
      Blog Post Author

      Hello Horst,

      • To deploy your Keystore, go to your Node Explorer at the bottom left of Eclipse -> Right Click on the Node Explorer ( The Folder ) --> Deploy Artifacts and select  Keystore.

               

      • The Service: CXF-endpont-IFLMAP-hcibsp is visible under your Tenant Management Node --> Component Status View. Select the TM Node in Eclipse on the bottom left of Eclipse --> and then check under Component Status View

               

      Regards,

      Bhavesh

      Author's profile photo Horst Müller
      Horst Müller

      Hello Bhavesh,

      Thanks for your reply!

      Now I'm able to deploy - unfortunatly it throws an error:

      /wp-content/uploads/2016/09/p1_1041426.png

      If I look into the "Tail-Log", I can see the following:

      /wp-content/uploads/2016/09/p2_1041427.png

      "...Request to worker node failed javax.net.ssl.SSLPeerUnverifiedExeption: peer not authenticated...."

      Do you have any hints on this?

      (Highly appreciated)

      Regards Horst

      Author's profile photo Bhavesh Kantilal
      Bhavesh Kantilal
      Blog Post Author

      Hello Horst,

      2 things to check,

      1. Have you imported the HCI Tenant's Public Certificate chain into your keystore? This is required to be done. This is the Learning#2 in this blog of mine.
      2. Is your HCI Key Pair signed by a CA in this list . This is as per Learnin#3 of this blog.

      Regards,

      Bhavesh

      Author's profile photo Former Member
      Former Member

      Bhavesh,

      When you say " Download BaseCertificate from your HCI URL" , are you talking about the runtime URL we get in the email after subscription of HCI ?

      Thanks

      Author's profile photo Former Member
      Former Member

      Bhavesh

      I have created a key pair , generated CSR, got and imported CA cert with root and intermediate certs.

      Imported HCI runtime Certs into keystore and deployed .

      Iam getting following error. Looks like permission error.

      2017 01 05 21:58:12#+00#ERROR#com.sap.esb.monitoring.cxf.connection.monitor.impl.CxfConnectionMonitor##anonymous#AvatarThreadLOW51#na#avphcin#p0578tmn#web##Request to worker node failed com.sap.esb.monitoring.cxf.connection.monitor.impl.CxfConnectionException: Request to https://XXXXXX.hana.ondemand.com/cxf/index.html returned 401 Reason: Unauthorized

      at com.sap.esb.monitoring.cxf.connection.monitor.impl.CxfConnectionMonitor.checkConnection(CxfConnectionMonitor.java:178)

      at com.sap.esb.monitoring.cxf.connection.monitor.impl.CxfConnectionMonitor.getComponent(CxfConnectionMonitor.java:79)

      at com.sap.it.nm.core.agent.ComponentSnapshotProducerTask.invokeAndGetComponents(ComponentSnapshotProducerTask.java:161)

      at com.sap.it.nm.core.agent.ComponentSnapshotProducerTask.produceSnapshot(ComponentSnapshotProducerTask.java:180)

      at com.sap.it.nm.core.agent.ComponentSnapshotProducerTask.produceSnapshot(ComponentSnapshotProducerTask.java:40)

      at com.sap.it.nm.core.agent.AbstractSnapshotProducerTask.call(AbstractSnapshotProducerTask.java:46)

      at com.sap.it.nm.core.agent.AbstractSnapshotProducerTask.call(AbstractSnapshotProducerTask.java:19)

      at com.sap.it.nm.core.concurrent.types.TaskContextImpl.call(TaskContextImpl.java:260)

      at java.util.concurrent.FutureTask.run(FutureTask.java:262)

      at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)

      at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)

      at java.lang.Thread.run(Thread.java:807)

      |

      com.sap.esb.monitoring.cxf.connection.monitor.impl.CxfConnectionException: Request to https://XXX.....us2.hana.ondemand.com/cxf/index.html returned 401 Reason: Unauthorized

      at com.sap.esb.monitoring.cxf.connection.monitor.impl.CxfConnectionMonitor.checkConnection(CxfConnectionMonitor.java:178)

      at com.sap.esb.monitoring.cxf.connection.monitor.impl.CxfConnectionMonitor.getComponent(CxfConnectionMonitor.java:79)

      at com.sap.it.nm.core.agent.ComponentSnapshotProducerTask.invokeAndGetComponents(ComponentSnapshotProducerTask.java:161)

      at com.sap.it.nm.core.agent.ComponentSnapshotProducerTask.produceSnapshot(ComponentSnapshotProducerTask.java:180)

      at com.sap.it.nm.core.agent.ComponentSnapshotProducerTask.produceSnapshot(ComponentSnapshotProducerTask.java:40)

      at com.sap.it.nm.core.agent.AbstractSnapshotProducerTask.call(AbstractSnapshotProducerTask.java:46)

      at com.sap.it.nm.core.agent.AbstractSnapshotProducerTask.call(AbstractSnapshotProducerTask.java:19)

      at com.sap.it.nm.core.concurrent.types.TaskContextImpl.call(TaskContextImpl.java:260)

      at java.util.concurrent.FutureTask.run(FutureTask.java:262)

      at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)

      at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)

      at java.lang.Thread.run(Thread.java:807)

      Author's profile photo Alexander Aigner
      Alexander Aigner

      Hello, have you resolved this? I have the same issue?

       

      Thanks and best regards,

      Alex

      Author's profile photo Alexander Aigner
      Alexander Aigner

      I think i have found the issue. It seems to appear for these two cases:

      1. The Key Pair was signed by a CA, but the CA is not in the list of supported CAs.
      2. You miss at least one of the CAs certificates that was used to sign your key pair

      In my case, it was the first. After testing with the SAP Passport as mentioned above, it was working.

       

      Thanks

      Author's profile photo Ravinder Boda
      Ravinder Boda

      Hi Bhavesh,

       

      Please can you show how to configure end to end Certificates between ERP - HCI - SF.

      We are using web dispatcher as well.. Not clear on how to configure certificates

       

      Author's profile photo Former Member
      Former Member

      Hi,

      Can you explain how to use this method to add the Internal SSO CA to the Keystore for internal testing purposes?

       

      Regards,

      Bradley Fernandes

      Author's profile photo Manu Bhutani
      Manu Bhutani

      Hi Bhavesh,

      In our tenant keystore i can see hcicertificate as keypair imported by SAP. 

      1. For SFTP we have created a new keypair with alias id_rsa. Is this fine?
      2. If not then how to go about it, if yes then do we need to get this id_rsa signed by CA for sftp interfaces.

      Regards,

      Manu

      Author's profile photo Pavan G
      Pavan G

      Hi Bhavesh,

      Thanks for the detailed blog.

      I am trying to deploy the known_hosts file to my CPI tenant to establish the connection between SFTP and CPI.

      But I am getting the below error:

      “Deploy artifact failed with error: You are not authorized to perform this operation”

      I am having the admin roles.

      Please let me know if there I am missing anything.

      Regards,

      Pavan