Technical Articles
How to sniff SAP CPI and SAP PO requests
As no one can resist a cute puppy and you are already here, let us use your attention to learn something new today!
Today’s blog post is about how HTTP(S) requests from SAP PO and SAP CPI can be recorded aka “sniffed” and the analyzed. But before we dive into the practical implementation, let’s take a little look at the “why”. Why should you be interested in recording the requests and when does such a feature benefit you?
So in the following minutes we will discover and answer the following topics.
Table of Contents
- When and why to sniff HTTP(S) requests
- If the request goes to the Internet
- If the request targets the local network
- Conclusion an advices
When and why to sniff HTTP(S) requests
If I have to address a new API in a SAP CPI or SAP PO interface, I usually test this API in advance via Postman or SoapUI and familiarize myself with it. If I have found out in the above mentioned tools how the call (which content? which url parameters? which header? which HTTP verb?) works, I rebuild it in the SAP interface.
Most times things go well, but sometimes the apparently completely identical call in SAP CPI/SAP PO does not work. In this case, it’s hard to find out what’s wrong, because theoretically the call should be identical to the one in the working Postman/SoapUI example. When sniffing the results done in Postman/SoapUI and SAP CPI/SAP PO with a neutral tool and then comparing the results, you can find out the differences and trace down the issue.
In addition to such situations, it can be helpful to record the requests to answer questions such as the following:
- Does the used SAP PO adapter add some additional header?
- Does SAP PO/SAP CPI use the headers I set up in the interface before?
- From which IP address does my request reach the recipient? (This may help to find out the routing and your public IP in case you are using a proxy server or a complex network infrastucture.)
- What exact requests are done, when using OAuth credentials in CPI or the OAuth handler(s) of SAP PO adapters and how does their content/headers look? (Real-world example)
Now that the purpose of sniffing is clear, let’s see how this works in practice.
If the request targets the internet
In case your target API/service is hosted and reachable via internet, we can trace the requests by online sniffing tools. The one I prefer (mainly because its ease of use) is called requestcatcher.com.
To sniff any request go to requestcatcher.com and choose a unique name for your session. (Be careful when setting this, because everyone knowing the session name can see your requests too. Thus it’s a good idea to use anonymized data for your tests and use a random session name. You could generate one e.g. with random.org tools.)
When you have entered a session name, you will be redirected to the requestcatcher-userinterface as shown in the screenshot above. Copy the interface url (1) and set it as target url in you SAP PO or SAP CPI receiver adapter or SAP CPI credential pair (or anywhere else where a webservice url can be configured.
Then run your interface/resend a message on or trigger your receiver adapter the way you like. As soon as SAP PO/SAP CPI do their requests, they will appear on requestcatcher’s interface as shown below.
Note: If you ran your interface and no requests shows up, you have a clue that something with your network settings/firewall/proxy might be wrong.
That’s it. Now you can trace down, what your SAP CPI/SAP PO interface is really sending. With this knowledge you hopefully can identify problems way faster, when setting up the next web service.
If the request targets the local network
The example above made use of the internet service “requestcatcher.com”. But what, if you target service is placed in your local network. Testing the requests with an internet-based sniffer wouldn’t be a good idea, since when directing your SAP PO to the internet, the request may go a total different way through your network, probably hopping over multiple proxies which could change your request. So when it comes to sniffing a locally targeted request, I suggest to use a local sniffing tool.
Again there are a lot of tools, but I personally had very good experiences with Fiddler Everywhere and therefore I will only go into this in the following.
- Download Fiddler Everywhere (It’s available on Windows, Mac OSX and Linux)
- Run the setup
- Launch Fiddler
Fiddler acts as a proxy server which sniffs for you the traffic. The advantage? You configure Fiddler as proxy into your SAP PO channels, but keep the original target. That way you can test and at the same time inspect your complete traffic flow.
After launching Fiddler for the first time, you have to setup Fiddler for use as SAP PO proxy. Therefore go to the settings page, write down Fiddler’s port and activate “Allow remote computers to connect”.
Then head over to your SAP PI channel and configure Fiddler as a proxy. The proxy ip address is the ip address of your computer/the computer Fiddler runs on. (Open CMD/terminal and type ipconfig /all on Windows or ifconfig on Linux/OS X to find our your computer’s ip address.) The proxy’s port is the one you wrote down from the Fiddler settings page.
Then (re-)run your interface. If you have correctly setup everything, the request of your SAP Po should occur in Fiddler’s request view on the left side. (See screenshot below.)
Clicking on a request, sniffed with Fiddler, brings up request and response details on the right-hand pane of the application. That’s it!
Conclusion and Advices
Both presented tools have helped me in the past to solve tricky situations. But you should always keep the following things in mind.
- Always choose the sniffing tool according to the location of the web service you want to address. If the service is located on the Internet, choose a sniffing tool on the Internet, if the service is on-premise, host a local sniffing tool. This is the only way to ensure that the routing/network conditions can be realistically analyzed.
- The main purpose of such tests is to see if headers are set correctly or if the payload is encoded correctly. The actual content of the payload is often irrelevant. So whenever possible, remember to anonymize the payload with pseudo data to avoid sending sensitive data to sniffer tools. (In theory, these tools adhere to data protection guidelines, but in practice something that didn’t get there in the first place cannot be leaked. Data economy is a key term here).
Finally, I would like to know if and if so, which tools you use to analyze SAP CPI/SAP PO requests.
Note: I was pointed in the comments by Eng Swee that there’s another great article series on HTTP sniffing. If you are still interested in the topic of HTTP tracing, you should put Ariel’s series on HTTP tracing as next item on your “what to read”-list.
(Cover picture licensed under CC BY 2.0 by Steve Jurvetson)
Hi Raffael,
Thank you for the post! It is really useful in scenarios where some uncommon protocol is used. Sometimes you need to see the headers and suddenly, the answer becomes clear!
Could you use SAP Cloud Connector + Fiddler on the local network for sniffing CPI requests? What do you think the disadvantage would be?
Regards,
Fatih
Hi Fatih,
Fiddler in combination with CPI and Cloud Connector may not work, because when configuring a local target in CPI you can't add an additional proxy server. The only way I see, would be setting up your local computer (running Fiddler) in Cloud Connector with e.g. virtual hostname "fiddler". And then configure this virtual hostname as receiver domain/url in CPI's receiver channel.
But doing so, you would break the end to end communication and won't benefit from Fiddler's proxy abilities. So in best case you could sniff the request, but couldn't trace the full flow.
Hi Raffael,
Yes, I was thinking about setting it as a receiver. It only makes sense as a last resort in scenarios where you can't use an online mock tool.
Thanks for the blog again!
Hi Fatih,
have you seen Eng Swee's comment? He describes/points to an article that describes how to sniff requests, even if Cloud Connector is used.
Hi Raffael, Thanks for the ping! I should follow new posts not to miss good comments.
Hi Raffael,
I have been using https://www.mockable.io it gives a good option to also send request back.
Hi Daniel,
thanks for pointing me to mockable. This looks really interesting and as far as I could see, the free version should be sufficient for a regular/daily consultant's job.
It's great to compare notes and possibly learn from one another 😉
For me, the choice of sniffer tool would depend on the capability of the middleware in supporting usage of (forward) proxy, rather than the location of the target service.
For PI, most if not all its HTTP-based adapters support configuration of forward proxy. So in the past, I've used Fiddler and it works generally well in both PI to cloud and PI to on-prem scenarios.
For CPI, there is no forward proxy configuration part in its adapters. In the past, I've used an HTTP dumping service like http://ptsv2.com to capture the content of the HTTP request. I think Request Catcher works in a similar way. Unfortunately, this approach is limited because you cannot trace the behaviour of the actual target service and its response. Some of the more complex APIs require multi-step HTTP calls, so you get stuck after the first call.
After Ariel Bravo Ayala published his excellent blog series on HTTP tracing (which I strongly recommend to everyone), I now use mitmproxy in reverse proxy mode for tracing of CPI requests. I think Fiddler can be configured in reverse proxy mode but I can't recall if I have used it or not.
So for CPI to cloud - I use mitmproxy running (in reverser proxy mode) in the cloud (in CF trial). This has worked very well especially when debugging issues integrating with SuccessFactors Employee Central and LMS.
For CPI to on-prem (via Cloud Connector) - if you have Cloud Connector admin role, then you can set up your own Cloud Connector to connect to your CPI tenant, and have mitmproxy running on your own system in reverse proxy mode.
Hi Eng Swee,
thank you for your detailed comment. I always appreciate feedback, especially when it brings so much new input.
I did not know the tool “ptsv2.com” until now. But in fact it looks quite similar (in terms of functionality) to requestcatcher.com.
Thanks also for the hint to Ariel’s blog series. I had indeed missed it, otherwise I would probably have written this article here differently or not at all. I have now linked Ariel’s Posts at the end of my article. ?
Ariel's blog series, not Vadim's :p
Uhm… that happens, if you have another person in mind, while typing. But I guess Vadim could have written it, too. :p
I fixed the references in the comments and blogpost. Thanks for pointing that out.
Hi Raffael and all others,
the tip using an external endpoint like requestcatcher or a MITM Proxy like Fiddler came in handy lately where I needed to trace some communication against an OAuth service of Microsoft Azure Platform (Turned out to be an NWDS bug, where the client secret got erased every second save).
Anyways, as I'm a fan of standard / out of the box solutions: Is there a way within logging or XPI inspector to see (at least outgoing) HTTP requests? I don't even beg for FTP or others. HTTP would be enough.
Cheers
Jens
This should be possible with XPI inspector. Choose mode "XI channel" and select the used receiver channel. In mode configuration check the "HTTP Trace" checkbox and switch the drop-down from "Headers only" to "Full request".
Hi Raffael,
Better late than never I came across your great post! What I've been using so far was "requestbin.net". It can also be self-hosted by the way: https://github.com/mxcxvn/requestbin.net
I didn't know "requestcatcher.com" yet, but must say that I don't like it much. Main reason is that the display of the caught messages is designed very simply and the small text box is not very practical. Second reason is that it requires TLS encryption and therefore the installation of the root certificate on the sender, if not yet available.
Thank to Eng Swee's comment above I discovered "ptsv2.com". I think that one is awesome becaues it fullfills all my wishes, including the ability to require basic authentication!
Best wishes,
Philippe
Raffael Herrmann Philippe Addor Jens Schwendemann
Eng Swee Yeoh
I wanted to check regards X-dynaTrace in http header generated and sent to consumer application. Though headers are disabled at SAP CPI end due to this invalid chars the message is failed to consume at end application.
Flow looks like SAP CPI -> EMP (kafka) -> Consumer app(.NET)
Rajesh, I haven't come across this problem/header, as far as I know. I assume the header is introduced after CPI and before reaching the target. Dynatrace seems to be a kind of cloud firewall (https://www.dynatrace.com/platform/application-security/). So I suggest you investigate with the corresponding network responsible about this. I don't think it's related to CPI.
PS: Better open a new question instead of hijacking this blog post. 😉