HCI: Deciphering HCI Keystore
Background
SAP provides 2 major operating models for HCI Installations,
- SAP Managed Operating Model – Major tasks for tenant related administration is handled by SAP including management of Keystore & Known_Hosts for SSH / SFTP Connections.
- Customer Managed Operating Model – Major Tasks for tenant administration is handled by the Customer. This model is currently applicable only for SAP HCI Partner Edition.
As most HCI Installations are on SAP Managed Operating Model –
- Have you ever wondered what the KeyStore from HCI contains?
- How does SSH / SFTP connectivity occur?
If yes, read on..
As-Is HCI Tenant Setup
As we start this journey to understand the setup, key important notes on the HCI Tenant:
- No Keystore is Deployed
- No Known_Hosts is deployed ( For SSH / SFTP Connections )
Learning#1 – Component CXF-endpoint-IFLMAP-hcibsp
Let’s start our journey of discovery with the component CXF-endpoint-IFLMAP-hcibsp. Have you looked at the Components under your Tenant Management Node? Wondered what the Component: CXF-endpoint-IFLMAP-hcibsp does?
As per SAP Documentation, the role of this Component is to simulate an external SSL Call every 30 seconds to your HCI Tenant Run-time node from the Tenant Management Node via the SAP HCI Load Balancer. The status of this SSL call helps HCI confirm if the runtime node is up and running.
So what happens when you do not have a Keystore deployed on your HCI Tenant? Well, this Component goes into error with the message: “Cannot connect, no Keystore deployed” as SSL connections require a Keystore.
Create your Keystore – Use a Self-Signed Certificate
Well, let’s just go ahead and create our Keystore then, we thought. In our case it was a “Test” tenant and as we were keen to get our hands dirty on a “Fresh” HCI Tenant, we thought lets create a Self-Signed Keypair and deploy on the Keystore.
The steps to create the Keystore are described in this blog: Starting with hana cloud integration-keep this in mind . In our case we used a Self-Signed Certificate and deployed the same on the Keystore. This is what our Keystore looked like, i.e., just a single Self Signed Keypair,
Post deployment of the Keystore, the status of CXF-endpoint-IFLMAP-hcibsp changed to “javax.net.SSLPeerUnverifiedExcepton: peer not authenticated”
Learning#2 – Add HCI Tenant’s SSL Certificate to your Keystore
The SAP documentation clearly states a SSL call is made to the HCI Load Balancer which led us to believe that the SSL call was failing as the HCI LoadBalancer’s SSL Certificates were not trusted in the Keystore. This brought us to the next step, where we added our HCI LoadBalancers Certificate Chain to the Keystore.
Download BaseCertificate from your HCI URL
Download Intermediate Certificate from your HCI URL
Download Root Certificate from your HCI URL
Import all 3 Certificates into your Keystore. This is what our Keystore now looked like.
Deploy your Keystore with the hope that the error vanishes!
Learning#3 – Self-Signed Certificates – Sorry Not Accepted
Alas, the error continued!
We had the feeling at the back of our mind that Self Signed Certificates would probably not be accepted in the Keystore. SAP provides a list of CA’s whose Certificates are accepted by the Load Balancer for authentication which meant we had to now get our certificate signed by a Trusted CA as listed on SAP documentation here.
Update & Deploy Keystore with a Trusted CA
The Keystore was then updated and deployed with a Keypair signed by a TrustedCA.Subsequently the error disappeared, and the CXF-endpoint-IFLMAP-hcibsp turned into status Green 🙂
SSH / SFTP, Known_Hosts and Your Keystore
So, what role does the Keystore play when you need to deal with SSH/SFTP Connections? How do you generate a known_hosts file?
In the case of a SAP Managed Operating Model, these complexities are hidden to HCI Developers but what does SAP do behind the scenes?
Prerequisite:
The SFTP Server IP Address is open in the SAP Firewall. This action had to be taken by SAP and a ticket to SAP is the only way to go about the same.
Test your SSH Connection
Use the Test Outbound Connection as described in the blog: HCI: Testing Outbound Connections from HCI to Test your connection to a SSH SFTP Server. The server returns the error “Retrieving known.hosts from cloud storage failed due to KeyStoreNotFoundException: Keystore with name: ‘known.hosts’”. The error clearly states – known_hosts is not deployed.
Learning#4 – Creating known_hosts file
A known_hosts file is a list of SFTP Servers and their public keys. This file enables your HCI Tenant know the list of Hosts that a SSH / SFTP Connection is allowed to.To generate a known_hosts file execute the below command from a SSH Terminal. In my case I use cygwin
Command: ssh-keyscan -t rsa <<IPAddress/HostName of SFTP Server>>
Copy the output of this command to a Text File and then deploy the same on the server.
SFTP Authentication & Keystore
So, now that our known_hosts is deployed let’s try a test connection we said. Alas, the error changed this time with error: “Auth Fail”. Our SFTP Server had been set up for Key Based Authentication with the exact KeyPair that was loaded into our Keystore as described in the wiki: Generating SSH Keys for SFTP Adapters – Type 2
So why is our SFTP Server not authenticating us when the Key is available in the Keystore? How does HCI know which Keypair to be used for SFTP Authentication we wondered considering that there is no field to provide the Alias Name of your Private Key.
HCI does not provide you with an option to provide a Private Key Alias name for Key Based Authentication for SSH / SFTP. Instead HCI looks for Key’s with alias: id_rsa or id_dsa in the Keystore and uses this to authenticate itself.
Learning#5 – Update Keystore to have a entry for Keypair with id_rsa
To avoid any impact to existing scenarios, we copied the existing Keypair entry into a new entry called “id_rsa” and deployed the keystore.
Now when the SSH connection is tested, it works like a charm!
Conclusion for the HCI KeyStore
- Ensure your Keypair is signed by a Trusted CA. ( No Self Signed Certificates )
- Ensure your tenant’s Load Balancer Certificates are loaded into your Keystore.
- Ensure a known_hosts file is created for SFTP Connections.Continue to append any new server to existing file.
- For SFTP / SSH ensure your KeyPair has the alias: id_rsa or id_dsa
Hi Bhavesh,
Very nice blog and informative. Thanks for sharing.
One thing to confirm since you brought up the Operating Model for HCI Installation. I know that first one (managed by SAP), the second one is not that I am aware. Does this mean that there's an on-premise installation for SAP HCI and is released already to customers and available for vendors? Or this is related to Netweaver 7.5 (SAP PO) which has the functionality on using the integration content of SAP HCI?
Cheers,
R-jay
Hello Ray,
This has nothing to do with an OnPremise SAP PO 7.5 and the ability to deploy Cloud Integration Content.There is no on-premise Installation of HCI.
The HCI Installation continues to be on the Cloud. Its like any normal HCI Instance on the Cloud. The Customer Managed Operating Model is available in the in HCI Instances under the license model "SAP HCI Partner Edition".
As this is a Partner Instance of HCI on SAP Cloud, SAP lets all Administration activities like deployment of keystores to be done by the Partner itself unlike the SAP Managed Model.
You may read more on this in link and its subsections: SAP HANA Cloud Integration
Regards,
Bhavesh
Hi Bhavesh,
Clear for me now. Thank you!
Cheers,
R-jay
Thanks Bhavesh for sharing !!!! this will solve lot of issue and doubts around secure connections !!!
Thanks for Information Bhavesh. Very Informative!
Bhargav
Hello, Thank you for that nice and helpfull blog post. I have a question about how to sign a certificate with a trustet CA like goDaddy.
I uploadet my CSR to goDaddys webpage but now I am stuck because I have to do the steps desribed under the following link:http://help.secureserver.net/article/7452?locale=en
Thanks a lot for your answer.
Hello Stefan,
we had the same issue with GoDaddy. Instead we took one of our existing key pair on another server and loaded the same in this key store as this was just used to make sure that the connection for internal HCI load balancer works.
if you are in a hurry to get your HCI tenant operational one tip-
Regards
Bhavesh
Hello Bhavesh
Thanks a lot for your answer. I tried it and it works. At the Moment I stuck with the next failure message but I still work on it. 😉
At the Moment I really don't Know How to Setting up our Webdispatcher and the HANA Cloud Connector. If I do the Connection test in C4C I'll get the following message in the HCI:
Inbound processing in endpoint at /COD/ERP/SimpleConnect failed with message "Fault:Could not send Message.", caused by "SunCertPathBuilderException:unable to find valid certification path to requested target"
But do you Know How does it works if you really want to sign a certificate? Is there any possibility to sign the certificate by SAP with a OSS Call or something like that?
A lot of questions, thank you for your further help.
Regards, Stefan
Hello Stefan,
Is your Tenant under a Partner Managed Model or SAP Managed Model? If SAP Managed Model, the keystore is controlled by SAP and SAP will generate the Keys for you. If partner managed, in my case we just used an existing key pair and left it at that as it has no impact to the other scenarios.
From the error it looks like, the message has arrived at HCI from C4C. When HCI tries to deliver this message to your ERP tentant the same fails with a SSL error.
This can be a extensive topic and hence would suggest you raise a question on the PI forum and I would be happy to assist there but prior to that couple of checks,
Regards,
Bhavesh
Hello Bhavesh,
our HCI is running under "customer managed operational model" and I'm tring to set up a connection ERP - HCI - C4C (and back).
I've created a keystore, created a CSR and received a certificate (using Keystore Explorer)
Could you explain where and how the keystore could be imported. I can't find the component "CXF-endpont-IFLMAP-hcibsp" you described.
Thanks for your Response!
Horst
Hello Horst,
Regards,
Bhavesh
Hello Bhavesh,
Thanks for your reply!
Now I'm able to deploy - unfortunatly it throws an error:
If I look into the "Tail-Log", I can see the following:
"...Request to worker node failed javax.net.ssl.SSLPeerUnverifiedExeption: peer not authenticated...."
Do you have any hints on this?
(Highly appreciated)
Regards Horst
Hello Horst,
2 things to check,
Regards,
Bhavesh
Bhavesh,
When you say " Download BaseCertificate from your HCI URL" , are you talking about the runtime URL we get in the email after subscription of HCI ?
Thanks
Bhavesh
I have created a key pair , generated CSR, got and imported CA cert with root and intermediate certs.
Imported HCI runtime Certs into keystore and deployed .
Iam getting following error. Looks like permission error.
2017 01 05 21:58:12#+00#ERROR#com.sap.esb.monitoring.cxf.connection.monitor.impl.CxfConnectionMonitor##anonymous#AvatarThreadLOW51#na#avphcin#p0578tmn#web##Request to worker node failed com.sap.esb.monitoring.cxf.connection.monitor.impl.CxfConnectionException: Request to https://XXXXXX.hana.ondemand.com/cxf/index.html returned 401 Reason: Unauthorized
at com.sap.esb.monitoring.cxf.connection.monitor.impl.CxfConnectionMonitor.checkConnection(CxfConnectionMonitor.java:178)
at com.sap.esb.monitoring.cxf.connection.monitor.impl.CxfConnectionMonitor.getComponent(CxfConnectionMonitor.java:79)
at com.sap.it.nm.core.agent.ComponentSnapshotProducerTask.invokeAndGetComponents(ComponentSnapshotProducerTask.java:161)
at com.sap.it.nm.core.agent.ComponentSnapshotProducerTask.produceSnapshot(ComponentSnapshotProducerTask.java:180)
at com.sap.it.nm.core.agent.ComponentSnapshotProducerTask.produceSnapshot(ComponentSnapshotProducerTask.java:40)
at com.sap.it.nm.core.agent.AbstractSnapshotProducerTask.call(AbstractSnapshotProducerTask.java:46)
at com.sap.it.nm.core.agent.AbstractSnapshotProducerTask.call(AbstractSnapshotProducerTask.java:19)
at com.sap.it.nm.core.concurrent.types.TaskContextImpl.call(TaskContextImpl.java:260)
at java.util.concurrent.FutureTask.run(FutureTask.java:262)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:807)
|
com.sap.esb.monitoring.cxf.connection.monitor.impl.CxfConnectionException: Request to https://XXX.....us2.hana.ondemand.com/cxf/index.html returned 401 Reason: Unauthorized
at com.sap.esb.monitoring.cxf.connection.monitor.impl.CxfConnectionMonitor.checkConnection(CxfConnectionMonitor.java:178)
at com.sap.esb.monitoring.cxf.connection.monitor.impl.CxfConnectionMonitor.getComponent(CxfConnectionMonitor.java:79)
at com.sap.it.nm.core.agent.ComponentSnapshotProducerTask.invokeAndGetComponents(ComponentSnapshotProducerTask.java:161)
at com.sap.it.nm.core.agent.ComponentSnapshotProducerTask.produceSnapshot(ComponentSnapshotProducerTask.java:180)
at com.sap.it.nm.core.agent.ComponentSnapshotProducerTask.produceSnapshot(ComponentSnapshotProducerTask.java:40)
at com.sap.it.nm.core.agent.AbstractSnapshotProducerTask.call(AbstractSnapshotProducerTask.java:46)
at com.sap.it.nm.core.agent.AbstractSnapshotProducerTask.call(AbstractSnapshotProducerTask.java:19)
at com.sap.it.nm.core.concurrent.types.TaskContextImpl.call(TaskContextImpl.java:260)
at java.util.concurrent.FutureTask.run(FutureTask.java:262)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:807)
Hello, have you resolved this? I have the same issue?
Thanks and best regards,
Alex
I think i have found the issue. It seems to appear for these two cases:
In my case, it was the first. After testing with the SAP Passport as mentioned above, it was working.
Thanks
Hi Bhavesh,
Please can you show how to configure end to end Certificates between ERP - HCI - SF.
We are using web dispatcher as well.. Not clear on how to configure certificates
Hi,
Can you explain how to use this method to add the Internal SSO CA to the Keystore for internal testing purposes?
Regards,
Bradley Fernandes
Hi Bhavesh,
In our tenant keystore i can see hcicertificate as keypair imported by SAP.
Regards,
Manu
Hi Bhavesh,
Thanks for the detailed blog.
I am trying to deploy the known_hosts file to my CPI tenant to establish the connection between SFTP and CPI.
But I am getting the below error:
“Deploy artifact failed with error: You are not authorized to perform this operation”
I am having the admin roles.
Please let me know if there I am missing anything.
Regards,
Pavan