Technical Articles
Do not log Payloads on CPI with MPL – use Persist Message instead
As an integration consultant and SAP Trainer (CLD900) I see many developers using groovy scripts to store the payload to the MPL (message processing logs).
This is not a reliable approach to log payloads in production tenants and should be used for testing and special purpose only, not as a general approach.SAP has added a circuit breaker to stop writing MPL attachments if 1 GB data is stored in 24 hours, see SAP note 2593825.
Use the correct integration pattern instead: Persist Message! This puts the payload (incl. headers and attachment(s) into the message store (database) for 90 days.
As SAP does not (yet) provide a UI to open those payloads from the tenant management node, we are delivering the tool for free in the mean time: WHINT MessageStore Viewer.
Feel free to download it and deploy on your CPI tenant within minutes.
Update: The MessageStore Viewer can be used to access CPI messages in SAP Cloud Platform (Neo) or in PO 7.50 (via SAP Cloud Connector), SCP on Cloud Foundy is not (yet) supported, as the OData API is not released for CF…
Hi Adam,
Good warning. I guess it has been the only simple place to view content.
Does this work as Raffael Herrmann writes about in his blog
https://blogs.sap.com/2020/01/04/how-to-read-write-sap-cpi-datastore-from-groovy/
Then I would guess it makes more sense to create some modification of the script to store the payload there.
Hi Daniel,
Storing the full message (payload, header and attachments) is a standard feature here.
The only missing part is the UI to access the message store (which is just another place in the database). It does not relate to the data store, but is easily accessible from the OData API.
I would usually recommend to stay in the standard as much as possible (as always in the SAP world). Using a (free) add-on is acceptable here, as it is not invasive (at iFlow runtime), but simply providing a read access after processing. In addition, I am pretty sure this will be offered by SAP in the standard UI...
Agreed it is best to stay at standard published interfaces.
It is not like you have the ability to make changes if SAP make a modification in next months release.
Thank you for this, I tried to persist messages before, but was unable to retrieve them 🙂
Do you know of any size limits when using the message store?
As a side note, it was SAP who recommended to use the groovy scripts when we started with CPI (as you probably know). Only later did they restrict the storage to 1 Gb.
Dear Martin, I am not aware of any restriction per message, but there is (currently) an overall disk space limit of 32 GB for each tenant...
That certainly still makes us have to think about the amount of messages (and thus data) we want to store in the message store, considering that we run into the 1 Gb limit daily and the fact that the messages are kept for 90 days in the message store.
Thanks, Adam.
WHINT service is running and connected to the tenant. I added a Persist shape at the start of the process to catch the incoming request. However, this is not showing in the WHINT monitor for some reason. Is there a delay in when the persisted messages who up in the monitor?
Dear Iddo,
usually you see all messages from the tenants MPL in the monitor (with and without persist steps).
There is no delay (also no additional persistence). Do you get an error message?
Best regards,
Adam
P.S.: you can also drop me an email under a.kiwon@whint.de and we can have a look together 🙂
I got it: you really need to search for the message ID. I was expecting to see the latest messages on top.
Hi everyone,
I would like to give clarification about a statement in the blog.
The above statement is not recommended by SAP. More specific, using a store for audit data to store monitoring data and pattern to put monitoring data in the store for auditing data is not recommended.
Here is the reasoning:
Therefore, if you exceed that quota, no persist operation would be possible and the message processing will be interrupted until the storage resources are available again.
Here is the documented reference for the persist messages permanently. As of now it is not possible to see how much of that quota is used for what.
Now, as the persist step is intended for audit data (more critical than the monitoring data), and there is no quota enforcement and no circuit breaker, as it is designed for audit data.
The circuit breaker in the message log is to protect your integration scenarios from filling the storage resources.
What you can do is carefully consider what you really need to store and when. For example, you can consider storing an id with which you can identify the message in a backend system.
Best regards,
Mitko (SAP Cloud Platform Integration Development)
Dear Mitko,
My understanding of "The message storage feature is useful for auditing purposes" (SAP Help) is exactly this. I see many companies having the requirement to persist a message (for auditing purposes), e.g. storing an incoming payload before it gets converted, transformed and sent to a receiver system/application.
Many consultants do this by storing the payload into the MPL, whereas the MessageStore is the correct place (even storing headers and attachments and not only the payload).
Of course this should not be used in each iFlow and only when there is a clear need (or even legal requirement) to do so, but storing the id to identify the message in a backend system can only be a weak workaround and applies only to the most simple scenarios.
I also see this 32GB limitation as a risk, but only until SAP will manage the first escalations from key accounts (customers). Then you will agree, that there might be customers with a high throughput who require more space than others.
I would recommend to either improve the storage behind the Persist Step or to remove it completely to avoid the confusion of an entire community.
If you like, I am open to discuss this further (ideally with Andreas Q. and Mandy K.), please drop me an e-mail under a.kiwon@whint.de .
Best regards,
Adam
Hello Adam,
if a message ends with an error, the message doesn't get stored by the persist step, whereas MPL does log it anyway. I reckon this is quite a downside of this approach.
Regards.
Cristian
Hi Cristian,
Agree, this is known and does not help when performing troubleshooting.
I think meanwhile I would go for recommending MPL logging, but only when archiving is activated. Out of the box SAP does not provide anything, so I guess we (whitepaper.id as an SAP-Partner) will have to build the proxy, and the UI to display the archived messages from the DMS. Stay tuned.
Best regards, Adam