Technical Articles
Blog series: HTTPS tracing and debugging – A simple way
Intro
Let’s face it, tracing and debugging HTTP connections (and specially HTTPS) can be tricky. It is usually your last traccing option, generally attempted when you have exhausted all other possible alternatives.
In fact, this is a question I frequently get from colleagues and clients. Those who have a background in SAP XI/PI may be able to remember some traces using the popular “TCPGateway” (TCPGw) that even SAP has recommended in some notes (i.e. note 856597). And despite it served its purpose back in time, it is no longer an alternative due to its major limitations such as the inability of handling encrypted connections or connection outside our network (Yes, you guessed… cloud connections).
Currently, there are different ways and programs to make HTTP traces, each with its own limitations and characteristics. In general terms, they are not very friendly and they require a deeper knowledge of networking or the protocol. Some of these tools are free, however, others more user-friendly are paid tools. (e.g. Wireshark, Tcpdump, Fiddler, Npcap, SSLSplit, etc).
In this series of posts I will present a simple way to trace and debug HTTP connections, using proxies that act as “Man in the middle”.
For those of you who…Wait, did you say “Man in the middle”? Isn’t that dangerous? Well, if you’re on an open and free network, let’s say in an airport, about to pay with your credit card, of course it is!. But in this case, what we want is a controlled use of a proxy that allows us to capture connections and then re-transmit them.
For those of you who don’t know it, let me introduce you to “mitmproxy” (Man-In-The-Middle-Proxy). According to their creators:
“mitmproxy is your swiss-army knife for debugging, testing, privacy measurements, and penetration testing. It can be used to intercept, inspect, modify and replay web traffic such as HTTP/1, HTTP/2, WebSockets, or any other SSL/TLS-protected protocols. You can prettify and decode a variety of message types ranging from HTML to Protobuf, intercept specific messages on-the-fly, modify them before they reach their destination, and replay them to a client or server later on. “
With this tool, not only we will be able to trace our middleware integration, but also any kind of application that uses HTTP(S) connections (Odata Services, Gateway, XI/PI, CPI, SOAMANAGER, The cloud connector, Eclipse, onwards and upwards).
This series of blog posts will cover the following topics:
- On-prem to on-prem
- On-prem to cloud
- Cloud to on-prem
- Cloud to cloud (using SAP Cloud Foundry)
- Final words, Q&A and snippets
To begin with, a few brief requirements:
- Eagerness to learn
- A trial SAP Cloud Foundry account
- Download mitmproxy according to your platform (plus Linux binaries)
- Access to a Cloud Connector (preferably yours and running locally)
- Some HTTP-based services to try out.
- Recommended:
- A SAP CPI instance to run some test
- A SAP PI system to test
- A local NodeJS installation (to run an HTTP echo server)
- Postman
- SOAP UI
That’s all. I hope that this series of post can be correctly understood and enjoyed as it has a lot of practical value. As the main focus is the proxy operation, I wont be giving much details about the endopoints such as the web services I’ll be using. However, shall you have any doubt, leave it in the corresponding comment section.
Let’s get started!
1st scenario: On-prem to on-prem
Ariel Bravo Ayala
Hi Ariel
Wow!! You certainly started the new year with a bang, with all these goodies for the rest of us.
I use Fiddler a lot but mostly in the regular proxy mode for on-prem to on-prem/cloud. It’s definitely good to learn another tool.
I enjoyed learning about how it could be used as a reverse proxy, and I especially love the idea of using it on Cloud Foundry – for me, that is the killer takeaway from this blog series.
The cloud-to-cloud tracing has always been a challenge in the past. I used to redirect the output of receiver HTTP-based channels to an HTTP server (like http://ptsv2.com/). This doesn’t always work well as it just logs the HTTP request, but do not mimic the HTTP response of the actual target endpoint. Sometimes for OData endpoints, the adapter also does some validation on the target endpoint and fail the processing before even sending out the HTTP request.
This would have come in really handy a few months back when I was trying to debug an issue with the SuccessFactors OData V4 adapter. The error in CPI was very cryptic and the tracing in CPI doesn’t provide the actual HTTP response, so it was a bit hard to figure out the issue – which in the end was just some missing values in the query parameter. If I was able to see the HTTP traffic, both request and response, the issue would have been identified much early on.
Anyway, I’m really happy now that I have my own cloud-based proxy to use at will.
Thanks again, really great work and big thumbs up!
Eng Swee
Hi Eng,
Thank you very much! I appreciate your recognition. I had a great time working on this and I honestly believe this is something useful to learn and understand.
For the cloud to cloud, I personally use the mitmproxy, running on a Linux instance that I have hosted in AWS, for me is faster as I have my own laboratory already in place. I’ve included the CloudFoundry approach as I think is simpler for those that do not have an online Linux sandbox.
Funny fact, I also had an issue with the odata adapter a few months ago that I traced with a reverse proxy. Basically the odata adapter attempts a connection twice: first for getting the metadata and the second connection to do the actual request. This is the reason that the odata adapter currently (although I’ve not rechecked this) does not work with json payloads as it first captures the metada in xml, it changes the Content-Type and it kills the request. I found this behaviour using the proxy.
Best regards,
Ariel
Yes, using Cloud Foundry lowers the entry requirement for setting this up as we can use SAP Cloud Platform trial for it - real nice of you to consider that. Great that you have your own setup hosted on AWS, would love to have one myself but unfortunately the exchange rates aren't that great at the moment! 🙁
Yes, I also noticed the first call to fetch metadata when sniffing the on-prem PI SFSF adapter. It's bizarre why it does it before every API operation, and their CPI counterparts behave the same way - when we look at the OData Audit Logs in SuccessFactors, it's littered with these metadata calls and this really causes a hit on the performance as SF metadata is huge (~5MB) because there are so many entities. 🙁
You can get a free tier in AWS ? There is no need in getting a monstrous instance. A single processor, 512MB RAM would do the trick.