CPI attachment downloader - IFlow version

[benjaminnehring]

Introduction

Do you need to download a lot of Message Processing Log (MPL) attachments? There are already some tools available that come to your rescue here. Please have a look into the solutions from Ivo Vermeer, Oğul Kaan Mert and Fatih Pense. The question that I had when reading the aforementioned blogs was: Can we implement such a functionality in CPI itself?

Specification

The tool here is an IFlow to which you send a set of query parameters in the http body and the response will be a gzip compressed tar archive (a .tar.gz file) of all attachments matching those query parameters. An example http request body looks as follows:

<query>

	<iflowName>log_as_mpl_attachment</iflowName>

    <regEx>.*[12]$</regEx>

	<logStart>2022-05-08T20:50:00</logStart>

	<logEnd>2022-05-10T13:00:00</logEnd>

    <timeZone>Europe/Berlin</timeZone>

</query>

This means that you want to download all attachments with a name matching the regular expression regEx belonging to messages of the IFlow iflowName which fall into the time interval between logStart and logEnd. The timestamps logStart and logEnd have to be in the format yyyy-MM-dd'T'HH:mm:ss and belong to time zone timeZone. A list of valid Strings for the timeZone field can be obtained when you execute TimeZone.getAvailableIDs() in a Groovy script.

Before deploying the IFlow (which I have developed on my CPI trial account) you need to populate two configuration parameters in it:

1. For cpi_host you need to insert the host name of the CPI tenant you want to download the MPL attachments from (the same host name that your Browser displays when you work in the WebUI on that tenant).


2. For CPI Cred you need to insert the name of the user credential which is used to call the MPL API of the CPI tenant configured in step 1.

When the IFlow is deployed and started you can try it out and send the above described request to the IFlow's http endpoint. If you have hit on Send and Download or on Save Response → Save to a file in Postman, Postman will suggest storing the file iflowName_attachments.tar.gz somewhere on your local machine. It contains the downloaded MPL attachments. The file names within the tar archive adhere to the following convention:

<TimeStamp>_<MessageGuid>_<AttachmentName>.txt

Implementation

The basis of the implementation is the MPL API which is documented in the API Business Hub. What is a bit hidden in the API response is the URL path to the attachment's payload. Dhinesh Kumar R has explained how to find it.

The IFlow first retrieves all message IDs of all messages of the IFlow iflowName which fall into the time interval logStart to logEnd. These are then used to obtain the attachment information for each message which then finally enables us to download each attachment's payload and create a tar archive out of it which we send gzip compressed as the http response. I don't go into the implementation much further here as the IFlow uses mostly standard features except for a small Groovy script which is used for filtering the attachments based on their name and for time stamp conversion (because the MPL API works with UTC time stamps).

A limitation of the current approach is certainly that its not possible to download several GB of attachment data this way as the tar archive is created within CPI's memory. To prevent out of memory errors there is a router step which checks the total payload size of all attachments and which will not start the download process if this reaches some threshold value. You might want to try out what are the limits here. I leave it for the interested reader as an open task to build a pagination mechanism to avoid this problem.

Please adjust the IFlow to your needs. For example if you want to group the attachments by message ID into separate folders in the archive, just replace the file name in the last Gather step with

${property.MessageGuid}/${property.TimeStamp}_${property.Name}.txt

Another use case of this IFlow may be for archiving purposes by replacing the http sender channel with a timer start event and by sending the .tar.gz file to a SFTP server. But for message archiving in CPI there is now also a standard solution available.

Conclusion

I developed this IFlow out of curiosity and had some fun along the way, so I decided to share it here. Now that I have talked so much about MPL attachments, I should probably give a warning: Be aware of the circuit breaker 🙂