image credit: user:TheDigitalArtist / pixabay.com under CC0
In a relatively short time we’ve taken a system built to resist destruction by nuclear weapons and made it vulnerable to toasters.
-- Jeff Jarmoc
Any sufficiently advanced bug is indistinguishable from a feature.
-- Rich Kulawiec, with apologies to Arthur C Clarke
The Problem
Internet security protocols and cryptography mechanisms are regularly being replaced by more complex schemes, as existing ones are regularly being proved vulnerable to cracking and hacking. From a practical perspective, this means that business IT departments are waging a constant battle to upgrade, patch, monitor, and reconfigure every system with any user interaction at all, not to mention any system that comes anywhere close to touching the Internet.
Which in the age of cloud computing and connected business is almost every system, isn’t it?
I don’t need to tell you that this is an expensive problem, requiring constant and increasing investment in time, materials, and skillsets. I probably also don’t need to tell you that the consequences of not making this investment are even more expensive.
Instead, I’ll relate a little story of how attempting to keep up with the latest and greatest can confound even the most well-intentioned of us.
Just sit right back and you’ll hear a tale…
image credit: user:Boskampi / pixabay.com under CC0
The Setup
SAP has a number of business connectivity solutions, middleware to allow our ERP systems to talk to vendors, suppliers, customers, and so forth up and down the proverbial value chain. One of these solutions, one acquired long ago from WebMethods, is the now slightly obscure SAP Business Connector. When I talk about Business Connector outside my organization, I regularly hear comments like, “Oh, I thought that was discontinued years ago; wasn’t it replaced by Process Integration?”
Not dead yet.
Business Connector is alive and well, although it’s true it may be approaching a final sunset sometime in the next few years. Meanwhile, however, SAP is supporting Business Connector at least through the end of 2020, and they are regularly supplying patches and updates for the latest (and only supported) version, 4.8.
And my employer still utterly depends upon Business Connector. We use it to electronically transmit purchase orders created in our SRM system to a handful of our most-used vendors. However, like many end-user organizations, we are constantly understaffed and overworked, so this is one system where we have tended to take an attitude of, “If it ain’t broke, don’t fix it.”
Not anymore.
We were not on the latest and greatest patch level of Business Connector, and yes, it had a few creaky joints, but it worked, so we left it alone. When the IETF (Internet Engineering Task Force) announced in 2015 that SSL (Secure Sockets Layer) version 3.0 was now to go the way of 1.0 and 2.0 into the graveyard of desupport, to be replaced by TLS (Transport Layer Security) of any version, we reviewed our patch notes and checked our configuration, and we saw that we were good. When the PCI Council (Payment Card Industry) recommended to their members earlier this year that TLS 1.0 should likewise be discontinued by the end of June, leaving only 1.1 and 1.2 in support, we still saw that we should be fine.
Vendors have responded to the PCI recommendation in different ways. Some simply deprecated TLS 1.0, while others elected to support only 1.2. Some told their customers in advance, and some… didn’t.
Some became even more strict than this, but more on this in a bit.
The Failure
So imagine our surprise when, as of July 2nd (the first business day of the month), all of our purchase orders for a particular vendor began to fail, and that vendor had said nothing about a change in protocol, and we had made no change to our system.
TLS 1.2, it turns out, is more finicky than its predecessors. It is far less tolerant of configuration errors, and it is far more common for such errors to exist, on either side of the transaction. Many vendors, for instance, may be misinterpreting the IETF and PCI statements to assume that anything with the letters “SSL” in it should be de-supported, and thus are turning off support for otherwise perfectly acceptable cipher suites. Others are incorrectly implementing the handshake protocol, and thus mistakenly rejecting well-formed “client_hello” messages because the “client_hello” is supposed to start with an SSLv3 announcement before requesting TLSv1.2.
And, the tech support departments at these vendors may not be able to tell you what’s wrong when you call them to ask why they’re rejecting your connections. You may hear bland statements such as, “We only support strong ciphers.” That’s nice, but telling us which strong ciphers would be nicer, as that way we could ensure we’re supporting them, too.
Despite nominal support for TLS 1.2 in Business Connector 4.8 as of CoreFix 9 Hotfix 1 (our patch level when this journey began), it turns out not to be as simple as that. We were running Business Connector on SAPJVM 5, and that version of Java was not, in fact, capable of full TLS 1.2 support. So that was problem number one.
We updated our system, implementing Service Release 13 (see Note 2510018), CoreFix 12 (see Note 2594583), and SAPJVM 7 at the latest patch level (see Note 1249462). We ensured we were running unlimited strength JCE (Java Cryptography Extension) policy files (see Note 1240081). We double-checked that all our vendor certificates and CA roots were up-to-date.
No joy. We continued to get the dreaded error:
iaik.security.ssl.SSLException: Peer sent alert: Alert Fatal: handshake failure.
The Resolution
Solving SSL handshake failures can be tricky, because by design the server does not return to the client any details about why they were rejected. Perhaps doing so could be construed as a security risk, revealing too much information to would-be attackers probing your system. However, that doesn’t seem to be much of a hindrance, as there are other ways to find out what protocols and ciphers are supported, as we’ll soon see.
We asked our vendor if they could find anything in logs at their end to indicate a reason for rejection, but we were unable to ever see such logs. We told them how we were configured, including cipher suites, and asked if they could see anything wrong. None of this helped.
In the end, the most useful tool was the website https://www.ssllabs.com, and their Test Your Server tool. Entering our vendor’s XML receiving system’s URL into the tool, the result told us something interesting.
Our vendor only supported a very restrictive, very high-end set of cipher suites. At a minimum, they required Perfect Forward Secrecy (PFS), which is a cryptographic protocol relying upon one-time keys, keys which are generated uniquely for each session and never reused. This means that even if in the future, when more powerful computers are available for brute-force cracking encryption keys, a recorded session cannot be decrypted just by possession of a compromised private key for one of the parties.
The most common PFS protocols used with TLS are Diffie-Helman key exchanges (DHE-RSA or DHE-DSA), and even stronger are elliptic curve Diffie-Helman exchanges (ECDHE-RSA, etc).
Business Connector supports Diffie-Helman PFS key exchanges over TLS 1.2, though not by default. This has to be specifically enabled. Business Connector does not, however, support elliptic curve key exchanges (as of this writing, I’m not aware of any SAP product which does).
Step 1: enable PFS.
This is done via Extended Settings in the SAPBC administration tool. Manually create the key watt.security.ciphers.pfs.dhe=true and save it. After doing so, the following new cipher suites are enabled:
TLS_DHE_DSS_WITH_AES_128_CBC_SHA256
TLS_DHE_DSS_WITH_AES_128_GCM_SHA256
TLS_DHE_DSS_WITH_AES_256_CBC_SHA256
TLS_DHE_RSA_WITH_AES_128_CBC_SHA256
TLS_DHE_RSA_WITH_AES_128_GCM_SHA256
TLS_DHE_RSA_WITH_AES_256_CBC_SHA256
We did so, and due to our ssllabs testing, we were now certain we had a commonly-supported cipher suite between us, but our orders still failed during handshake.
Step 2: enable SNI.
There are various extensions to the TLS protocol which servers and clients may or may not support, and one of these is Server Name Indication (SNI). This is a facility for websites which have multiple hosts (or hostnames), each with their own security certificate, sharing the same IP address. This might be the case, for instance, if the website is a virtual host (such as in VMware, or in a cloud hosting scheme).
The problem with this arises during a TLS or SSL handshake, however, as the client doesn’t typically mention the hostname it is trying to connect with until the HTTP(S) header is presented, and the handshake occurs prior to presenting of the HTTP header. Thus, the server has no idea which hostname the client is requesting, and doesn’t know which certificate to send to the client.
SNI solves this by giving the client a way to indicate the hostname as part of the handshake request (the “client_hello” message). If the server supports SNI, it can read this extension and present the correct certificate. If the server requires SNI, then this has to be explicitly enabled on the client.
Fortunately, Business Connector supports SNI, though again it is not enabled by default. This is managed via SSL Client Settings, in the field SSL Extensions. By default, you will see signature_algorithms here, and nothing else. By default, Business Connector (as a client) will send two extensions: renegotiation_info and signature_algorithms. These cannot be turned off, so even if you remove signature_algorithms from the SSL Extensions setting, it will be sent.
The solution in this case was to replace signature_algorithms with server_name.
Success! Our SAPBC client now sent all three extensions, the handshake completed, and we made contact! Or, rather, sent a purchase order.
One Vendor’s Fix is Another Vendor’s Failure
Enabling server_name with the problematic vendor solved one problem, but it introduced another. Our Business Connector client now required that servers support SNI, and not all of them do (nor do they all need to). Fixing the problem for one vendor broke things for another.
Fortunately, Business Connector supports two variants of SNI: essentially one where we require SNI, and one where we support (but do not require) it. To switch to this second variant, we replaced our SSL Extension setting with server_name(noncritical), and voila!
Problem solved.
image credit: user:typographyimages / pixabay.com under CC0
Anatomy of a Successful SSL/TLS Handshake
With SSL Debug enabled in Business Connector, this is what a successful TLS 1.2 handshake with PFS and SNI looks like in the server log:
ssl_debug(168): Starting handshake (iSaSiLk 5.106)...
ssl_debug(168): Sending v3 client_hello message to <vendor URL>:443, requesting version 3.3...
ssl_debug(168): Sending extensions: renegotiation_info (65281), signature_algorithms (13), server_name (0)
ssl_debug(168): Received v3 server_hello handshake message.
ssl_debug(168): Server selected SSL version 3.3.
ssl_debug(168): Server created new session 71:58:74:E4:B3:49:89:5A...
ssl_debug(168): CipherSuite selected by server: TLS_DHE_RSA_WITH_AES_256_CBC_SHA256
ssl_debug(168): CompressionMethod selected by server: NULL
ssl_debug(168): TLS extensions sent by the server: renegotiation_info (65281), server_name (0)
ssl_debug(168): Server supports secure renegotiation.
ssl_debug(168): Received certificate handshake message with server certificate.
ssl_debug(168): Server sent a 2048 bit RSA certificate, chain has 3 elements.
ssl_debug(168): Received server_key_exchange handshake message.
ssl_debug(168): Server sent a 1024 bit DH key
ssl_debug(168): Received server_hello_done handshake message.
ssl_debug(168): Sending client_key_exchange handshake (1024 bit DH)...
ssl_debug(168): Sending change_cipher_spec message...
ssl_debug(168): Sending finished message...
ssl_debug(168): Received change_cipher_spec message.
ssl_debug(168): Received finished message.
ssl_debug(168): Session added to session cache.
ssl_debug(168): Handshake completed, statistics:
ssl_debug(168): Read 5000 bytes in 6 records, wrote 377 bytes in 4 records.
SSL version 3.3 is the technical version number for TLS 1.2. Don’t be confused by this. TLS 1.0 was the same as SSL 3.1. See, SAP isn’t the only vendor out there who likes to rename and renumber things.
And never, ever, give up.
Useful Notes
1094412: Release and Support Strategy of SAP Business Connector 4.8
2510018: Business Connector 4.8 Service Release 13 (SR13)
2594583: SAP Business Connector 4.8 Core Fix 12
2537910: Business Connector Developer 4.8 CoreFix 7
1249462: How to import an SAP JVM patch into an SAP BC
1240081: Java Cryptography Extension (JCE) Jurisdiction Policy Files
2316176: SAP Business Connector 4.8 and TLS 1.2
1157248: SAP Business Connector with SSL / Common Problems
- SAP Managed Tags:
19 Comments
