Skip to Content
Technical Articles
Author's profile photo Philippe Addor

Cloud Integration: Message Headers and Properties Transfer Behavior

Retained or not retained? That’s the question.

When it comes to asynchronously passing messages between IFlows using a Data Store or JMS Queue on Cloud Integration, I was often scratching my head when trying to remember in which case headers or properties are passed on to the receiver IFlow. Sometimes they get retained and therefore passed on, sometimes they get lost and not passed on.

In the synchronous case of forwarding a message to another IFlow, using Process Direct Adapter, it’s less confusing as it’s more simple: Headers are retained, properties lost. For the sake of completeness, I have also added the synchronous cases at the end of the table.

But the asynchronous case, where you could use the Data Store or JMS Queues, this is not so straightforward as header parameters and properties are treated differently depending on the technique used to store or forward messages in between processing by multiple IFlows.

This post aims to provide an overview serving as a quick reference for developers designing or developing IFlows. The first part is about the asynchronous cases (temporarily persisting data and picking them up from another IFlow) and is split up in two parts, one for Data Store and one for JMS Queues. The second part is for the synchronous cases.

The table lists if a writing step or receiver adapter retains some of the data and on the other hand if a reading step or sender adapter makes the data available again on the receiver’s Integration Process side.

 

Activity

Header

Properties

Remarks

Asynchronous Messaging

Data Store Steps/Adapter

Save in Data Store (Write step)

retained (if configured in channel)

lost

Configuration needed to retain Header.

Read any message from Data Store (=> get all entries in bulk) – Select Operation

lost

lost

see 1)

Read specific message from Data Store (by specifying the entry ID) – Get Operation

available

lost

DS Sender Adapter

available

lost

Remember to list the headers in the field Allowed Headers of the integration flow’s runtime configuration.

1) This step creates a bulk (multi-) message with a messages node and a message node per message (with attribute id containing the message ID). Therefore it’s logically not possible to retain headers/properties for every message in the actual message properties of the bulk message. If you need this, use the Data Store Sender Adapter instead that creates single messages per DS entry and keeps the headers!

JMS Adapter

JMS queue write – JMS receiver channel

retained

retained (if configured in channel)

Configuration needed to retain Properties.

JMS queue read – JMS sender channel

available

available (if configured JMS receiver channel)

Remember to list the headers in the field Allowed Headers of the integration flow’s runtime configuration.

Note: Headers and exchange properties defined in the integration flow before the message is saved to the queue must not exceed 4 MB in total.


Persist Step

Save message with Persist step

retained

lost

Can only be read with external tools (like CPI Helper), therefore only for monitoring purposes.

Synchronous Messaging

Process Direct Adapter

Process Direct receiver channel

retained

lost

Process Direct sender channel

available

lost

Remember to list the headers in the field Allowed Headers of the integration flow’s runtime configuration.

HTTP and Kafka Adapters

Same behavior as with Process Direct.

 

Hope this is helpful for others as it is for me.

In case of questions or if you find any errors or inconsistencies, please let me know.

 

References

Data Store Operations:

https://help.sap.com/docs/CLOUD_INTEGRATION/368c481cd6954bdfa5d0435479fd4eaf/79f63a4bf5a44b5996aa34c51e2f187f.html

JMS Adapter:

https://help.sap.com/docs/CLOUD_INTEGRATION/368c481cd6954bdfa5d0435479fd4eaf/79edc04c91574a6bb8c15ae3e1a27b03.html

Data Store Adapter:

https://help.sap.com/docs/CLOUD_INTEGRATION/368c481cd6954bdfa5d0435479fd4eaf/0993f2aa14124376a4adc7c5ba95d3f8.html

 

Update: Blog updated as per Alexander Bundschuh’s comments

Update 09/20: Update header/property limit for JMS queues as per Jeremy Ma‘s comment.

 

Assigned Tags

      7 Comments
      You must be Logged on to comment or reply to a post.
      Author's profile photo Alexander Bundschuh
      Alexander Bundschuh

      Hi Philippe,

      thanks for putting this together but actually your findings wrt JMS are wrong, for JMS the header is retained and available, prerequisite is that the header names are maintained in the allowed header configuration, I have just updated the recipient list pattern of our integration flow design guidelines using the new dynamic JMS queue settings (guidelines not published yet) and can confirm that the headers are passed to the second integration flow when using JMS

      Can you please update your table

      Thanks and Regards

      Alex

      Author's profile photo Philippe Addor
      Philippe Addor
      Blog Post Author

      Hi Alex

      Thank you for reading my blog and your valuable feedback! You're right, I just tested it as well. I have missed to set the Allowed Header settings in this case because I wrongly assumed this is only necessary for direct calls like Process Direct or HTTP.

      By the way, the same seems to apply to the Data Store Sender adapter. When configuring the Allowed Headers, it also retrieves the headers.

      I have updated the table. And with this, it actually becomes quite consistent and much easier to remember. 🙂 Headers are retained, and properties only with JMS and according configuration. 

      The design guidelines that you're pointing to is the following page, right? Or will there be something new? Just to have it linked here.  https://help.sap.com/docs/CLOUD_INTEGRATION/368c481cd6954bdfa5d0435479fd4eaf/27c247e016184cee97581fbaa53359f7.html

      Best regards,

      Philippe

      Author's profile photo Alexander Bundschuh
      Alexander Bundschuh

      yes, that's the one, the flow however will be changed, with the dynamic queues I don't need the pass-through integration process any more

      Alex

      Author's profile photo Jeremy Ma
      Jeremy Ma

      Thanks for document the matrix Philippe.

      Two points to note here for JMS adapter:

      Cheers!

      Author's profile photo Philippe Addor
      Philippe Addor
      Blog Post Author

      Hi Jeremy,

      Thanks for your inputs. I would like to be a bit more specific here, see my comments in italics:

      • 4MB transfer limit for header/properties/msg.  So is recommend to send payload via body instead.
        PA: Agree to that, except that you also list "msg". The documentation however says that the limit is pertaining to header and properties, but no limit for message (which I can confirm based on my experience).

      • Uncheck ‘Transfer Exchange Properties’ in receiver JMS adapter which is also SAP’s recommendation.
        PA: I know that it's the recommendation, but am not sure why it is recommended :-). I as IFlow developer control and know what I store in the properties. Therefore I can make sure to limit the properties values to a few bytes (or max. 4 MB as stated above). I would for example avoid saving the payload in a property before sending it to JMS (or delete it before). Therefore I don't understand this recommendation and don't see the difference to the header properties which are always forwarded to JMS. What's your take on that? 

      Philippe

      Author's profile photo Jeremy Ma
      Jeremy Ma

      Hi Philippe,  Correct 4MB for header/properties but body is still limit by 40MB 3074668.

      With recent update to the JMS adapter w. compression you can dramatically reduce the body payload substantially when enabled.    I have supported a very large scale customer where they were exceeding over 500/tps and the payload size have direct impact to throughput.  Internally benchmark team validate performance w. payloads sized in body if msg only.

      Not every developer would be as diligent to delete/clean up their properties and next dev may not follow same hygiene.

      Author's profile photo Philippe Addor
      Philippe Addor
      Blog Post Author

      The note that you mention is about the configurable size limit of the HTTP and SOAP sender adapters. Are you sure this is related to JMS? Because in the above linked SAP Help document it clearly says "There are no size limits for the payload. The message is split internally when it is put into the queue."

      Agree to the point about diligence of different developers. 🙂