SAP Business Connector and the Increasing Complexity of Internet Security
image credit: user:TheDigitalArtist / pixabay.com under CC0
In a relatively short time we’ve taken a system built to resist destruction by nuclear weapons and made it vulnerable to toasters.
— Jeff Jarmoc
Any sufficiently advanced bug is indistinguishable from a feature.
— Rich Kulawiec, with apologies to Arthur C Clarke
Internet security protocols and cryptography mechanisms are regularly being replaced by more complex schemes, as existing ones are regularly being proved vulnerable to cracking and hacking. From a practical perspective, this means that business IT departments are waging a constant battle to upgrade, patch, monitor, and reconfigure every system with any user interaction at all, not to mention any system that comes anywhere close to touching the Internet.
Which in the age of cloud computing and connected business is almost every system, isn’t it?
I don’t need to tell you that this is an expensive problem, requiring constant and increasing investment in time, materials, and skillsets. I probably also don’t need to tell you that the consequences of not making this investment are even more expensive.
Instead, I’ll relate a little story of how attempting to keep up with the latest and greatest can confound even the most well-intentioned of us.
Just sit right back and you’ll hear a tale…
image credit: user:Boskampi / pixabay.com under CC0
SAP has a number of business connectivity solutions, middleware to allow our ERP systems to talk to vendors, suppliers, customers, and so forth up and down the proverbial value chain. One of these solutions, one acquired long ago from WebMethods, is the now slightly obscure SAP Business Connector. When I talk about Business Connector outside my organization, I regularly hear comments like, “Oh, I thought that was discontinued years ago; wasn’t it replaced by Process Integration?”
Not dead yet.
Business Connector is alive and well, although it’s true it may be approaching a final sunset sometime in the next few years. Meanwhile, however, SAP is supporting Business Connector at least through the end of 2020, and they are regularly supplying patches and updates for the latest (and only supported) version, 4.8.
And my employer still utterly depends upon Business Connector. We use it to electronically transmit purchase orders created in our SRM system to a handful of our most-used vendors. However, like many end-user organizations, we are constantly understaffed and overworked, so this is one system where we have tended to take an attitude of, “If it ain’t broke, don’t fix it.”
We were not on the latest and greatest patch level of Business Connector, and yes, it had a few creaky joints, but it worked, so we left it alone. When the IETF (Internet Engineering Task Force) announced in 2015 that SSL (Secure Sockets Layer) version 3.0 was now to go the way of 1.0 and 2.0 into the graveyard of desupport, to be replaced by TLS (Transport Layer Security) of any version, we reviewed our patch notes and checked our configuration, and we saw that we were good. When the PCI Council (Payment Card Industry) recommended to their members earlier this year that TLS 1.0 should likewise be discontinued by the end of June, leaving only 1.1 and 1.2 in support, we still saw that we should be fine.
Vendors have responded to the PCI recommendation in different ways. Some simply deprecated TLS 1.0, while others elected to support only 1.2. Some told their customers in advance, and some… didn’t.
Some became even more strict than this, but more on this in a bit.
So imagine our surprise when, as of July 2nd (the first business day of the month), all of our purchase orders for a particular vendor began to fail, and that vendor had said nothing about a change in protocol, and we had made no change to our system.
TLS 1.2, it turns out, is more finicky than its predecessors. It is far less tolerant of configuration errors, and it is far more common for such errors to exist, on either side of the transaction. Many vendors, for instance, may be misinterpreting the IETF and PCI statements to assume that anything with the letters “SSL” in it should be de-supported, and thus are turning off support for otherwise perfectly acceptable cipher suites. Others are incorrectly implementing the handshake protocol, and thus mistakenly rejecting well-formed “client_hello” messages because the “client_hello” is supposed to start with an SSLv3 announcement before requesting TLSv1.2.
And, the tech support departments at these vendors may not be able to tell you what’s wrong when you call them to ask why they’re rejecting your connections. You may hear bland statements such as, “We only support strong ciphers.” That’s nice, but telling us which strong ciphers would be nicer, as that way we could ensure we’re supporting them, too.
Despite nominal support for TLS 1.2 in Business Connector 4.8 as of CoreFix 9 Hotfix 1 (our patch level when this journey began), it turns out not to be as simple as that. We were running Business Connector on SAPJVM 5, and that version of Java was not, in fact, capable of full TLS 1.2 support. So that was problem number one.
We updated our system, implementing Service Release 13 (see Note 2510018), CoreFix 12 (see Note 2594583), and SAPJVM 7 at the latest patch level (see Note 1249462). We ensured we were running unlimited strength JCE (Java Cryptography Extension) policy files (see Note 1240081). We double-checked that all our vendor certificates and CA roots were up-to-date.
No joy. We continued to get the dreaded error:
iaik.security.ssl.SSLException: Peer sent alert: Alert Fatal: handshake failure.
Solving SSL handshake failures can be tricky, because by design the server does not return to the client any details about why they were rejected. Perhaps doing so could be construed as a security risk, revealing too much information to would-be attackers probing your system. However, that doesn’t seem to be much of a hindrance, as there are other ways to find out what protocols and ciphers are supported, as we’ll soon see.
We asked our vendor if they could find anything in logs at their end to indicate a reason for rejection, but we were unable to ever see such logs. We told them how we were configured, including cipher suites, and asked if they could see anything wrong. None of this helped.
In the end, the most useful tool was the website https://www.ssllabs.com, and their Test Your Server tool. Entering our vendor’s XML receiving system’s URL into the tool, the result told us something interesting.
Our vendor only supported a very restrictive, very high-end set of cipher suites. At a minimum, they required Perfect Forward Secrecy (PFS), which is a cryptographic protocol relying upon one-time keys, keys which are generated uniquely for each session and never reused. This means that even if in the future, when more powerful computers are available for brute-force cracking encryption keys, a recorded session cannot be decrypted just by possession of a compromised private key for one of the parties.
The most common PFS protocols used with TLS are Diffie-Helman key exchanges (DHE-RSA or DHE-DSA), and even stronger are elliptic curve Diffie-Helman exchanges (ECDHE-RSA, etc).
Business Connector supports Diffie-Helman PFS key exchanges over TLS 1.2, though not by default. This has to be specifically enabled. Business Connector does not, however, support elliptic curve key exchanges (as of this writing, I’m not aware of any SAP product which does).
Step 1: enable PFS.
This is done via Extended Settings in the SAPBC administration tool. Manually create the key watt.security.ciphers.pfs.dhe=true and save it. After doing so, the following new cipher suites are enabled:
We did so, and due to our ssllabs testing, we were now certain we had a commonly-supported cipher suite between us, but our orders still failed during handshake.
Step 2: enable SNI.
There are various extensions to the TLS protocol which servers and clients may or may not support, and one of these is Server Name Indication (SNI). This is a facility for websites which have multiple hosts (or hostnames), each with their own security certificate, sharing the same IP address. This might be the case, for instance, if the website is a virtual host (such as in VMware, or in a cloud hosting scheme).
The problem with this arises during a TLS or SSL handshake, however, as the client doesn’t typically mention the hostname it is trying to connect with until the HTTP(S) header is presented, and the handshake occurs prior to presenting of the HTTP header. Thus, the server has no idea which hostname the client is requesting, and doesn’t know which certificate to send to the client.
SNI solves this by giving the client a way to indicate the hostname as part of the handshake request (the “client_hello” message). If the server supports SNI, it can read this extension and present the correct certificate. If the server requires SNI, then this has to be explicitly enabled on the client.
Fortunately, Business Connector supports SNI, though again it is not enabled by default. This is managed via SSL Client Settings, in the field SSL Extensions. By default, you will see signature_algorithms here, and nothing else. By default, Business Connector (as a client) will send two extensions: renegotiation_info and signature_algorithms. These cannot be turned off, so even if you remove signature_algorithms from the SSL Extensions setting, it will be sent.
The solution in this case was to replace signature_algorithms with server_name.
Success! Our SAPBC client now sent all three extensions, the handshake completed, and we made contact! Or, rather, sent a purchase order.
One Vendor’s Fix is Another Vendor’s Failure
Enabling server_name with the problematic vendor solved one problem, but it introduced another. Our Business Connector client now required that servers support SNI, and not all of them do (nor do they all need to). Fixing the problem for one vendor broke things for another.
Fortunately, Business Connector supports two variants of SNI: essentially one where we require SNI, and one where we support (but do not require) it. To switch to this second variant, we replaced our SSL Extension setting with server_name(noncritical), and voila!
image credit: user:typographyimages / pixabay.com under CC0
Anatomy of a Successful SSL/TLS Handshake
With SSL Debug enabled in Business Connector, this is what a successful TLS 1.2 handshake with PFS and SNI looks like in the server log:
ssl_debug(168): Starting handshake (iSaSiLk 5.106)…
ssl_debug(168): Sending v3 client_hello message to <vendor URL>:443, requesting version 3.3…
ssl_debug(168): Sending extensions: renegotiation_info (65281), signature_algorithms (13), server_name (0)
ssl_debug(168): Received v3 server_hello handshake message.
ssl_debug(168): Server selected SSL version 3.3.
ssl_debug(168): Server created new session 71:58:74:E4:B3:49:89:5A…
ssl_debug(168): CipherSuite selected by server: TLS_DHE_RSA_WITH_AES_256_CBC_SHA256
ssl_debug(168): CompressionMethod selected by server: NULL
ssl_debug(168): TLS extensions sent by the server: renegotiation_info (65281), server_name (0)
ssl_debug(168): Server supports secure renegotiation.
ssl_debug(168): Received certificate handshake message with server certificate.
ssl_debug(168): Server sent a 2048 bit RSA certificate, chain has 3 elements.
ssl_debug(168): Received server_key_exchange handshake message.
ssl_debug(168): Server sent a 1024 bit DH key
ssl_debug(168): Received server_hello_done handshake message.
ssl_debug(168): Sending client_key_exchange handshake (1024 bit DH)…
ssl_debug(168): Sending change_cipher_spec message…
ssl_debug(168): Sending finished message…
ssl_debug(168): Received change_cipher_spec message.
ssl_debug(168): Received finished message.
ssl_debug(168): Session added to session cache.
ssl_debug(168): Handshake completed, statistics:
ssl_debug(168): Read 5000 bytes in 6 records, wrote 377 bytes in 4 records.
SSL version 3.3 is the technical version number for TLS 1.2. Don’t be confused by this. TLS 1.0 was the same as SSL 3.1. See, SAP isn’t the only vendor out there who likes to rename and renumber things.
And never, ever, give up.
1094412: Release and Support Strategy of SAP Business Connector 4.8
2510018: Business Connector 4.8 Service Release 13 (SR13)
2594583: SAP Business Connector 4.8 Core Fix 12
2537910: Business Connector Developer 4.8 CoreFix 7
1249462: How to import an SAP JVM patch into an SAP BC
1240081: Java Cryptography Extension (JCE) Jurisdiction Policy Files
2316176: SAP Business Connector 4.8 and TLS 1.2
1157248: SAP Business Connector with SSL / Common Problems
Hello Matt Fraser,
Very nice blog. We came across this situation but in PI SFTP adapter.
One of the sunny day we received communication from the ABC bank that - They have announced they will start support of the ECDH_NISTP256 and DH_GROUP_EXCHANGE_SHA256 cryptographic key exchange algorithms for transfers using secure file transfer protocol (SFTP) on so and so date. They will support both the current and new cryptographic key exchange algorithms until a date. At that date, they will stop supporting the old ones and we must have upgraded by then.
Another bank PQW has sent us a similar request, but they are only asking to us upgrade by a later date than the ABC bank . So, today PQW has successfully tested the downwards compatibility by sending an ACH file from an upgraded PI QAS system to the non-upgraded PQW bank testing environment.
In this situation, we need to accelerate the SFTP upgrade in PI prod. system along with Java Cryptography Extension (JCE) Jurisdiction Policy Files ... well PI system is not excluded from the situation 🙁
Specially SAP note 2344454 - "com.jcraft.jsch.JSchException: Algorithm negotiation fail" error in Message Monitoring while using SFTP Receiver Adapter - very handy
Ah, so the SFTP adapter for PI does now support Elliptic Curve key exchange. That's relatively new.
We've had the same situation with vendors, where they update their protocols or their certificates and give minimal notice, perhaps a few months, but perhaps only a few weeks, and in one case, no notice at all -- one day things just stopped working, which prompted this blog post.
Thanks for your reply!
I sat back and felt with you at each step of the journey.
Thanks for writing this. This will give everybody a hope when struggling in the middle of a domino effect.
Probably not the only area where priorities might change from one day to another just because something was changed outside of the area of your responsibility.
And a good example of an area where even a solid settled setup needs to be reviewed and reconfigured in depth from one day to another.
A good read for all the ones that do think "running simple" can be done by just scratching the surface of a topic. Always good to have someone around who is an expert or get's it when the going gets rough.
My SAPBC story ... I invested early in getting known to the SAP BusinessConnector in late 1999 ... wow more than 18 years ago.
Fascinated by the new mapping technology, fully browser based administration, having fun looping through XML and learning about the different industry standards for the messages.
And then I was immediately thrown into the coldest watered project in my consultant life .. Brrr.. freezing cold. 🙂
A small but long story .. but a success story:
The SAP BusinessConnector .. and it's amazing ROI over the years.
Looking forward to the 25th SAP BC anniversary party in ... 2024 😉
Thanks, Christian! Good to know there's someone else out there still playing around with this tool, as it seems to have been forgotten by most. We've always assumed it would eventually be replaced by PI, but as we've started dipping our toes into PI's waters, we can see that it's not necessarily any simpler there, and indeed may even be more complex. Time will tell, as we've barely begun our PI journey. With BC, however, this now represents, hmm let me see... about 16 years we've been using it. I think it was 2002 when we set up our first XML PO integration with a vendor.
So what should we do in 2024, assuming SAPBC truly is still alive then? Light fireworks at TechEd? 😉
Speaking of TechEd, looks like I will probably make it to Barcelona this year. Will you be there?
There are for sure some more 4.8ers around .. running with a low footprint for a single scenario .. or being the backbone of a mid'sized message scenario.
Not fancy new .. but running like a solar cell .. silently just doing it's work.
Until .. until "the other side" changes something 😉
BCN would be a good place for SAP BC aNniversary party.
Let's check the location at TechEd BCN together this year 😉
One of my customers use Business Connector extensively as well and it feels so good to know there are other folks who are also using Business Connector. Like I tell a lot of folks - our BC was not restarted in 3 years and when I came onboard and we wanted to patch the windows server - I was nervous, because heck I had never restarted it and was not sure if it would come back up. I lived to tell the tale 🙂
Indeed, you are here to tell the tale! I know well the nervousness about restarting a system that has not been touched for years -- what settings might have been changed and forgotten that will become active upon reboot? That could be at the Windows level or the application level: SAPBC certainly has some settings which become active immediately, and others which require a restart, just like Windows.
Thanks for your comment!
Been a while since you wrote, but once you do, they are always a great read!
I could totally relate to what you went through (albeit yours probably had a higher sense of urgency) – it reminded me of a similar SSL/TLS journey we went through with PI (yes, PI is not spared the headache of this). Loved the way you expounded on the various nuances of TLS 1.2.
– Eng Swee
Thanks for linking this. I wish I had found your blog earlier! Even though your experience was for PI and not BC, there are some definite similarities (and, also, we may be going through this with PI in the near future, too, but at least in this circumstance it will be for greenfield interface implementations and not rushing to fix existing interfaces that broke). I did see Markus' blog when I was researching our problem, but I didn't catch that the referenced SSL config file is inside the iaik_ssl.jar. I assumed this was something specific to PI that didn't exist for BC, but BC does have this same jar file (at least, it does as of CoreFix12; I don't think it was there for CoreFix9). I never thought to look inside the jar.
We did not engage SAP Support for our issue, as I wasn't sure for quite a long time that it wasn't really an issue with our vendor and not us. In the end it really was us, but I still fault the vendor for not advising us of their requirements and instead making us discover them like a hacker would, probing their systems. Also, in the end, it wasn't a bug within SAPBC, just a problem with our configuration. Once we understood how to properly configure it, the system worked as anticipated.
But, to your point (and mine): updating to newer versions of TLS is not so simple as putting a patch or hotfix on the system. And TLS 1.3 has been in draft form for a while; it can't be far off before it's released...
Good lord! I can see where having to keep up with all this, hunt down issues like this, stay "ahead of the hackers", could easily drive a man to drink. hahaha
You drive me to drink, and I'll buy the first round!
Our Cybersecurity Manager read through the blog, and her comment was, "Wow, I thought chasing down vendors for Windows patches was hard..."
But, I have to admit, a part of me finds all this stuff, the intricate interplay of all these protocols and cryptography suites and cipher keys, really fascinating. It's a major headache, but fascinating, nonetheless.
Yup, same sentiment for me on our PI "adventure" - learnt all this stuff the hard way, but it's not something you'll ever get sitting down in a classroom!
I thought our FTP to SFTP transition was painful enough but this is like the whole other level. Gee whiz.... Yes, "run simpe" can be anything but when you get down into the pits. Sadly, the folks who are actually in the pits running the whole operation rarely get the recognition they deserve. I hope some folks in an unnamed school district will read this. 🙂
Thanks for sharing!
It has been read by at least one manager here, and I think it's helped shed some light on why I kept looking at the team trying to change out all the printers on short notice and force me to work with them on that, while this was going on, as if they were stone cold crazy.
To be fair to SAP, they didn't invent these security protocols, and as far as I can tell they are doing what they can to keep up and support what needs supporting, but the world of Internet security and cryptography changes rapidly, and it's hard for anyone, or any organization, to keep up.
Now, why nuclear-grade cryptography should be necessary to send a simple purchase order... that I don't know. 🙂
any idea where can I find teh SSL Debug Logs ?
Sorry it took me a while to get back to you! First, you have to enable the SSL Debug logging in Business Connector itself by connecting to the web administration interface and navigating to Security -> SSL Settings. Then click Edit SSL Settings and toggle SSL Debugging to true. If I recall correctly, you have to restart Business Connector to enable the settings change.
Once you've done that, the SSL Debug logs are interweaved with the regular "server" logs, and you can find the most recent in the UI via Logs -> Server. However, you can also locate them in the filesystem on the BC server under <Installation Path>\Server\logs, where they will be the ones that start with server and then a date/time-stamp.