Chronicles of a PI TLS 1.2 upgrade
We recently upgraded our PI 7.4 systems to SPS 15. Nothing really fancy in terms of features since SAP development in the PI space has pretty much come to a standstill these days. However, as part of the upgrade, it included the new IAIK library that now supports TLS 1.2. As mentioned in my previous blog post over a year ago, this is one of the more popular issue/discussion last year as it is a key functionality. Fast forward one year later, and that is still one of the most viewed blog of mine since the migration to this new SAP Community platform, so my guess is this topic is still relevant today as it was last year.
As mentioned, this new library is delivered as part of SAP Note 2284059 – Update of SSL library within NW Java server and it provides a fair amount of details related to the configuration of the library. Another useful resource related to this is Markus Schalk‘s blog Outbound support for TLS 1.1/1.2.
In this blog, I will aim to share my own experience during this upgrade project with the intention of highlighting some of the key areas to take note for anyone going through a similar journey.
Upgrading the library
So once we have the IAIK library installed, we could begin testing our HTTPS interfaces. As mentioned in the SAP Note, SAP strongly recommends that testing is done for all connections prior to deployment in production.
We can verify the new IAIK library is being used and TLS 1.2 capability is available by using XPI Inspector to troubleshoot HTTP SSL connections. As shown below, the SSL debug logs shows that the client hello is sent requesting SSL version 3.3 (i.e. TLS 1.2). Additional observation is that the renegotiation_info and signature_algorithms TLS extensions are also included.
Once we began testing, we noticed java.net.SocketException: Connection reset errors on some of the interfaces.
One of our external partners confirmed that when using TLS 1.2, they also require the SNI (Server Name Indication) extension to be sent as well. Without this, the SSL handshake would fail and result in the error above.
SSL config file
The SAP Note mentions that the SSL handshake can be further configured using a custom SSL configuration file. So first of all, we need to find a copy of the default configuration file. This is important because there are many existing properties configured there, and if we start with a blank file instead, all these default properties would be missing.
Surprisingly, although the SAP Note refers to the SSL configuration file, it does not provide a copy of it, and neither does it mention where it can be found. Lucky for us, Markus has already gone through the exercise and mentioned that it can be found in the iaik_ssl.jar file in the PI system. However, the path provided by him was not applicable in our system (maybe because ours is dual stack), so I’m listing where I found it on my system instead:-
Once we have default config file, we configured SNI support by following section 5.5 of the note. Since our interfaces connect to both servers that require SNI as well as those that do not support SNI, we add the following property to the file.
Subsequently, this custom property file is placed in the PI system, and the VM parameters are configured to read this file.
The real fun begins!
After a system restart, we thought that this would have solved the problem, but on the contrary this was where the “fun really began”!
While some of the previously failing connections began to work, others began to show different errors.
Just to be sure that the configuration was done correctly, we ran another XPI inspector trace, and we can now see that the server_name extension is being included.
We noticed that the connections on non-SOAP adapters (like our Advantco DYNCRM adapter) were working fine, but the SOAP connections were facing a new error:-
Communication over HTTPS/PROXY with host verification. Unable to create a socket
As this was a unexpected and bizarre error, after troubleshooting the issue without much progress – guess what…. we decided to log an incident on the SAP Support Portal. Anyone who has experienced SAP Support would know what this means…. PING PONG match!
Despite having paid premium $$$ for SAP MaxAttention support for the duration of the upgrade project, it did not change the fact that the Incident ticket had to make its round amongst SAP’s First Level support before getting any traction. Dealing with SAP Support was in itself a test of patience, resolve, muscle flexing and anger management!
We were even sent on a wild goose chase, when one of them recommended that we add the property in the following format to the config file.
The issue with this was:-
- It wasn’t mentioned in the note, so we wouldn’t have known to configure it
- We can only list one server, what if we have more than one server requiring SNI support?
Of course, we being good SAP customers, followed the suggestion and
wasted spent our time testing it to no avail.
It was after some muscle flexing that we finally got through to the development team. And to everyone’s surprise, it was revealed that the above was an undocumented feature that was only meant for internal testing purposes. “So why did your colleague suggested it to us in the first place???!!!”
Excuse me, did you see my bug fix?
So, apparently from the SOAP adapter development expert, the host verification error was due to a bug fix (SAP Note 1381198) that was previously released but somehow did not make its way to the latest release. Imagine that – a bug fix got lost in SAP’s code repository…. why??? how??? Anyway, we were then asked to implement SAP Note 2413354 in order to get the fix for the issue.
With that in place, we were supposed to configure the following in the SOAP channels that were connecting to the servers requiring SNI support. The “promise” of course would be that this would fix our issue – guess what… it didn’t!!
99 little bugs in the code
So, after we implemented the above note, we hit the following new error in our SOAP channels
java.io.EOFException: Connection closed by remote host.
This issue particularly took a longer time to resolve because the error was appearing intermittently. At times the connection began to work (and we had a premature “hooray” moment), only for it to fail again later. We were unable to consistently reproduce the error and as such it was hard to SAP to determine the root cause. After a while, we finally noticed the following trend:-
|Channel had hostVerification = true||Messages fail with error java.io.EOFException: Connection closed by remote host.|
|Channel then had hostVerification parameter removed||Messages fail with previous error Communication over HTTPS/PROXY with host verification.|
|Channel reconfigured back with hostVerification = true||Messages start to process successfully!!|
|After some time elapsed||Messages fail back again with error java.io.EOFException: Connection closed by remote host.|
This seems to indicate some sort of temporary cache related issue. So I decided to troubleshoot the issue using Fiddler to analyse the network traffic. Fiddler acted as a listening proxy (mimicking our organization’s forward proxy) and captured the network traffic.
The analysis revealed that there is a difference in the following HTTP header related to proxy authentication.
Proxy-Authorization: Basic <Authentication>
|Channel had hostVerification = true|
|Channel then had hostVerification parameter removed|
It seems the fix in SAP Note 2413354 introduced a new bug where the proxy authentication HTTP header was no longer sent. This error was masked because our organization’s forward proxy caches the proxy authentication for a certain time limit. So when the hostVerification was removed, the proxy authentication was sent and cached.
We managed to confirm this behavior by engaging our network team to capture Wireshark traces of the traffic when we performed the above sequence of steps. Additionally, we also collected XPI Inspector trace using Example 100 as advised by SAP, which showed the details of the error.
With this, we managed to convince SAP that there was indeed a bug and therefore a fix was required.
However, it still took SAP a few iterations of private fixes before finally releasing the SAP Note publicly. This was because the development team was unable to test the scenario internally as they do not have a proxy server in their infrastructure (imagine that… SAP with all their billions, it must have all gone to Leonardo!) They had to rely on us to implement the private fix, perform the test and collect the Wireshark and XPI traces.
Anyway, finally we received SAP Note 2435720, and after thorough testing of all the interfaces, we were glad that this marked the end of our saga.
Upgrading the TLS library, whilst necessary is not a trivial pursuit, especially if there are a lot of HTTPS-based interfaces with external partners. As recommended by SAP, make sure everything is tested before you deploy it to production, if not, you never know if some nasty issue awaits you.
Your narration of the story is as funny as the details of the blog are enlightening. Great work Eng and thanks for documenting it for the benefit of the community.
Thanks Sanjeev. SAP Support provided all the fodder for the story 🙂
I second that. This was so much fun to read and anyone dealing with support and prio 1 incident must have been there provided being long enough in the SAP ecosystem.
Thanks for the valuable insights that came as a sidenode to the fun read. We currently dealing with SNI, TLS 1.2 in 2022, so this is still some valid blogpost of yours
BTW: For those who are looking for a next chapter: SAP still struggling with hostVerification: https://launchpad.support.sap.com/#/notes/3148408 😉
Throughout the Blog i really felt your pain, we went through the exact same issues, we managed to solve the issue around the negotiation extensions but after we ran into the next issue after that (we had already spent many hours attempting to debug the issues) we reverted the failing adapters to the SapJVM crypto libs from a previous release.
Your extensive documentation of the issues and fixes will probably prompt us to go back and fix the issues the correct way, many thanks for your effort!
Thanks for your comment. Totally understand what you must have went through. Hope this helps the second time round you try to upgrade the TLS library.
Thanks for the insights Eng, always great to read your blogs. I have had similar daunting experience, however due to time pressure we ended up with an alternative solution of up/down stepping TLS 1.2 calls to/from TLS 1.0 at the load balancer level which was in front of the PI system.
With TLS 1.3 is just around the corner, it will be interesting to see how promptly it is supported in PI ( and with how many bugs 🙂 ).
Thank you for your comment. It would be interesting to hear your experience too, especially how the SSL handshake can still be completed with the stepping up/down.
With regards to TLS 1.3, I think it's still in draft mode so it might take a while before it becomes widespread across the internet/industry. Furthermore, TLS 1.2 has been around since 2008, and it's only now that certain sites are enforcing it as a minimum, and I haven't seen any widespread attack on it (like Poodle or Heartbleed on SSL 3.0), that will cause everybody to panic yet. Of course, TLS 1.3 is great, but the reality is, I think it will take a while.
We are sorry that you were experiencing this issue. In order to get further help and assistance, please feel free to contact the Customer Interaction Center. They are happy to assist and available 24/7. All contact info for their phone numbers, email address, and more can be found here: https://launchpad.support.sap.com/#/notes/560499/E (SAP Note 560499).
Thank you for your comment and providing information of the CIC.
Appreciate if you can elaborate on:-
Any customer can contact CIC for assistance with non-technical queries such as existing incidents, questions about SAP ONE Support Launchpad and its applications, help for S-User queries. CIC is one of the interfaces between customers and the SAP Support organization and they are available 24 hours a day, seven days a week, 365 days a year. You can contact them any time.
We will review your feedback to ensure a better experience moving forward and we’re always striving to improve SAP Support processes.
Thank you for your clarification. Will definitely take note of that for future incidents. As for the mentioned incident, IMHO I don't think contacting the CIC would have been relevant.
Please can you tell me what sap jvm are you using?
I made all the steps, i tested from xpi_inspector and tls1.2 connection is working fine.
but from http_aae i get "javax.net.ssl.SSLHandshakeException: Server chose unsupported or disabled protocol: Unknown-3.3"
It's on PI 7.4, so SAP JVM 6 is used.
I suggest you try with other adapters like SOAP or REST. It's possible that HTTP_AAE adapter does not use the updated IAIK library - we didn't test for this as we don't use it.
Hi Eng Swee
For sure http adapter is using jvm ssl libraries. I had to update the jvm to latest patch of jvm 6 an now i can connect on tls 1.1. tls 1.2 is not supported by jvm 6 so i have to upgrade to 7.5
Hi Florin, Hi Eng Swee Yeoh,
We also want to connect on TLS1.2 to, for example, SalesForce, but our System is SAP NW7.4 SPS21, which still uses SAP JVM 6.1, and this does not Support TLS1.2.
We have to Upgrade to NW7.5 / SAP JVM 8.1, which will eventually allow TLS1.2 Connection, as standard.
I have tried the SSLContext.properties, but with no luck. Is there another Workaround, or upgrade is the only solution?
Thanks and Kind regards,
Hi. I am not being able to connect using REST. It fails at handshake after ‘sending extensions’. I got 'received alert message alert fatal internal error'.Do you have any idea what it could be? It seems the target uses SNI and basis has followed instructions on sap note 2604240 but we still receiver the same error. Thanks.
Hi Eng Swee
You know if once I have upgraded to TLS 1.2, can I continue connecting a web server that uses SSLv3 112 bits DES-CBC3-SHA?
I did the PI upgrade (7.40 SP 17) to support TLS 1.2, but in our testing environment I consume a web service from a site that has not yet been updated, and running the tests gives me the "Handshake Failure" error.
We are trying to reach a TSL 1.2 server but we are getting the same error as above. Did you manage to fix the above issue?
The problem I had was to be able to connect to a service with a previous security version and I solved it by adding the parameters maxSSLVersion and minSSLVersion in the channel module.
Hi Eng Swee,
I am facing a similar situation in my REST Receiver Adapter.
The exception which i receive is Exception reading SSL message: java.io.EOFException: Connection closed by remote host.
XPI Inspector Log says that the SSL Handshake is successfully completed.
We are on SP12 of PI 7.4 Dual Stack and TLS 1.2 version.
Can you kindly suggest a possible fix for this ?
Dear Eng Swee,
I am facing the issue is that: SAP PI ( NW731EXT_15_REL, Service pack 10) is in internal network, not connected to internet. We installed SAP Web Dispatcher (WD) in DMZ, and need connect from PI via WD to internet Rest API. After configuration, from SAP PI Server (OS) level, with installed SOAP UI/Postman i can send message to internet Rest API and get back response via WD. But from SAP PI Rest receiver channel, i got http 503 unavailable error. With some tracing, i saw that information package sent by SOAP UI, protocol are TCP and TLSV1.2, while information package sent by SAP PI Rest receiver channel are TCP/TLSV1 and SSLV2. These difference are issue? Then how can i fix it?
Thank you so much!
Hi Eng Swee/Khac Bao Anh Nguyen,
We are also facing some SSL-TLS version issues on PI 7.3.1 – duel stack issues. while posting data to FTP and MS SQL 2012 (jdbc channels)
Please suggest how we can resolve the issues, is there any work around. current system,we have TLS 1.0 only with JVM 1.6(045) version available. so to convert to TLS 1.2, is we need to upgrade patch or JVM upgrade is enough or updated is need for IAIK library ?
challenge here is we dint have any SAP support to contact or raise the OSS to get help