Technical Articles
Cloud Integration: Message Headers and Properties Transfer Behavior
Retained or not retained? That’s the question.
When it comes to asynchronously passing messages between IFlows using a Data Store or JMS Queue on Cloud Integration, I was often scratching my head when trying to remember in which case headers or properties are passed on to the receiver IFlow. Sometimes they get retained and therefore passed on, sometimes they get lost and not passed on.
In the synchronous case of forwarding a message to another IFlow, using Process Direct Adapter, it’s less confusing as it’s more simple: Headers are retained, properties lost. For the sake of completeness, I have also added the synchronous cases at the end of the table.
But the asynchronous case, where you could use the Data Store or JMS Queues, this is not so straightforward as header parameters and properties are treated differently depending on the technique used to store or forward messages in between processing by multiple IFlows.
This post aims to provide an overview serving as a quick reference for developers designing or developing IFlows. The first part is about the asynchronous cases (temporarily persisting data and picking them up from another IFlow) and is split up in two parts, one for Data Store and one for JMS Queues. The second part is for the synchronous cases.
The table lists if a writing step or receiver adapter retains some of the data and on the other hand if a reading step or sender adapter makes the data available again on the receiver’s Integration Process side.
|
Activity |
Header |
Properties |
Remarks |
|---|---|---|---|
|
Asynchronous Messaging |
|||
|
Data Store Steps/Adapter |
|||
|
Save in Data Store (Write step) |
retained (if configured in channel) |
lost |
Configuration needed to retain Header. |
|
Read any message from Data Store (=> get all entries in bulk) – Select Operation |
lost |
lost |
see 1) |
|
Read specific message from Data Store (by specifying the entry ID) – Get Operation |
available |
lost |
|
|
DS Sender Adapter |
available |
lost |
Remember to list the headers in the field Allowed Headers of the integration flow’s runtime configuration. |
|
1) This step creates a bulk (multi-) message with a messages node and a message node per message (with attribute id containing the message ID). Therefore it’s logically not possible to retain headers/properties for every message in the actual message properties of the bulk message. If you need this, use the Data Store Sender Adapter instead that creates single messages per DS entry and keeps the headers! JMS Adapter |
|||
|
JMS queue write – JMS receiver channel |
retained |
retained (if configured in channel) |
Configuration needed to retain Properties. |
|
JMS queue read – JMS sender channel |
available |
available (if configured JMS receiver channel) |
Remember to list the headers in the field Allowed Headers of the integration flow’s runtime configuration. |
|
Note: Headers and exchange properties defined in the integration flow before the message is saved to the queue must not exceed 4 MB in total.
|
|||
|
Save message with Persist step |
retained |
lost |
Can only be read with external tools (like CPI Helper), therefore only for monitoring purposes. |
|
Synchronous Messaging |
|||
|
Process Direct Adapter |
|||
|
Process Direct receiver channel |
retained |
lost |
|
|
Process Direct sender channel |
available |
lost |
Remember to list the headers in the field Allowed Headers of the integration flow’s runtime configuration. |
|
HTTP and Kafka Adapters |
|||
|
Same behavior as with Process Direct. |
|||
Hope this is helpful for others as it is for me.
In case of questions or if you find any errors or inconsistencies, please let me know.
References
Data Store Operations:
JMS Adapter:
Data Store Adapter:
Update: Blog updated as per Alexander Bundschuh’s comments
Update 09/20: Update header/property limit for JMS queues as per Jeremy Ma‘s comment.
Hi Philippe,
thanks for putting this together but actually your findings wrt JMS are wrong, for JMS the header is retained and available, prerequisite is that the header names are maintained in the allowed header configuration, I have just updated the recipient list pattern of our integration flow design guidelines using the new dynamic JMS queue settings (guidelines not published yet) and can confirm that the headers are passed to the second integration flow when using JMS
Can you please update your table
Thanks and Regards
Alex
Hi Alex
Thank you for reading my blog and your valuable feedback! You're right, I just tested it as well. I have missed to set the Allowed Header settings in this case because I wrongly assumed this is only necessary for direct calls like Process Direct or HTTP.
By the way, the same seems to apply to the Data Store Sender adapter. When configuring the Allowed Headers, it also retrieves the headers.
I have updated the table. And with this, it actually becomes quite consistent and much easier to remember.
Headers are retained, and properties only with JMS and according configuration.
The design guidelines that you're pointing to is the following page, right? Or will there be something new? Just to have it linked here. https://help.sap.com/docs/CLOUD_INTEGRATION/368c481cd6954bdfa5d0435479fd4eaf/27c247e016184cee97581fbaa53359f7.html
Best regards,
Philippe
yes, that's the one, the flow however will be changed, with the dynamic queues I don't need the pass-through integration process any more
Alex
Thanks for document the matrix Philippe.
Two points to note here for JMS adapter:
Cheers!
Hi Jeremy,
Thanks for your inputs. I would like to be a bit more specific here, see my comments in italics:
PA: Agree to that, except that you also list "msg". The documentation however says that the limit is pertaining to header and properties, but no limit for message (which I can confirm based on my experience).
PA: I know that it's the recommendation, but am not sure why it is recommended :-). I as IFlow developer control and know what I store in the properties. Therefore I can make sure to limit the properties values to a few bytes (or max. 4 MB as stated above). I would for example avoid saving the payload in a property before sending it to JMS (or delete it before). Therefore I don't understand this recommendation and don't see the difference to the header properties which are always forwarded to JMS. What's your take on that?
Philippe
Hi Philippe, Correct 4MB for header/properties but body is still limit by 40MB 3074668.
With recent update to the JMS adapter w. compression you can dramatically reduce the body payload substantially when enabled. I have supported a very large scale customer where they were exceeding over 500/tps and the payload size have direct impact to throughput. Internally benchmark team validate performance w. payloads sized in body if msg only.
Not every developer would be as diligent to delete/clean up their properties and next dev may not follow same hygiene.
The note that you mention is about the configurable size limit of the HTTP and SOAP sender adapters. Are you sure this is related to JMS? Because in the above linked SAP Help document it clearly says "There are no size limits for the payload. The message is split internally when it is put into the queue."
Agree to the point about diligence of different developers. 🙂