Skip to Content
Technical Articles
Author's profile photo Prasanth Rao

Handling Large Data with Content Enricher and OData v2 adapter

In this blog post, you will learn how to better model a Content Enrich scenario in SAP Cloud Integration. All full-scale integrations would inevitably have one or more Enrich steps. Very often, the adapter connected to the Content Enricher pulls in all the available data from the backend system causing crashes and performance degradation. The efficiency of the integration therefore greatly depends on the finer modeling of the Enricher step especially when large data is involved.

In this blog post, we will model a simple integration flow to handling large data is using dynamic queries using property and OData v2 $filter query option. In the example here, we will be enriching CompoundEmployee entity with FOLocation by creating a dynamic filter in Successfactors OData v2 adapter.

The Scenario

I have configured a looping process(as seen below) to fetch and enrich Compound Employee records in pages. The script step “Parse keys to Properties” has the logic for creating a dynamic filter.

Content Enricher configuration for this scenario is shown below.

Understanding Content Enrich Functionality

Content Enrich works by matching the key elements from Original Message(i.e. Compound Employee) to the key elements in Lookup Message(i.e. FOLocation) and then intelligently aggregating the snippets of the matching data from Lookup into the Original message.

We can always help optimize the data fetched in the Lookup Message by passing the appropriate key elements as $filter parameter for the adapter. In this example, we are comparing key element location from Compound Employee to key element externalCode and therefore we will define and configure a dynamic filter on externalCode for optimization.

Configuring Content Enrich to work with Large Data Volumes

Step 1: Defining the script

The main step.  Below is the Script Code for “Parse keys to Properties” to construct the dynamic filter. The first part of the script parses the Original Message and selects all the key elements using XmlSlurper. The second part forms the key element in the OData $filter format  and sets it to property named leadKey.

You can see that I always have the property configured with a value. Else there would be a runtime failure if the property is missing.

PS: “in” operator is unique to Successfactors. For standard OData services ( such as Hybris Marketing, XSOData etc), you will need to use the “or” operator instead.

import java.util.HashMap;
def Message processData(Message message) {
    def body = message.getBody(;
    //Parse the key from Original message
    def response = new XmlSlurper().parse(body);
    def keys = response.'**'.findAll{ node-> == 'location' }*.text();
    //Formulate the filter query. Eg: 	$filter=externalCode in '1710-2018'
    def leadKey = "'" + keys.unique().join("','") + "'";
    if(leadKey == null || leadKey.trim().isEmpty()){
        message.setProperty("leadKey", "");
    } else{
        message.setProperty("leadKey", "\$filter=externalCode in " + leadKey);
    return message;

Step 2: Configure the SucessFactors OData v2 adapter

The simplest step. Add the property leadKey in the query option. And your scenario is good to go!

Advantages of the approach

  1. Far lesser calls to the backend systems.
  2. Lesser and importantly relevant data only pulled in each enrich call.
  3. Overall improvement in throughput and execution times.

Points to consider

  • Only elements defined as filterable can be used in a $filter query. A good design approach could be to select only fields that are filterable as Lookup key elements. In case the required key element is not filterable, you can also consider another filterable element that is related to the selected key element to form the query.
<Property Name="externalCode" Type="Edm.String" Nullable="false" sap:filterable="true" sap:required="true" sap:creatable="false" sap:updatable="false" sap:upsertable="true" sap:visible="true" sap:sortable="true" MaxLength="32" sap:label="Code"/>​
  • API also can have limit restrictions. For instance, the ‘in’ operator in Successfactors only supports 1000 values in a call. You may need to have a looping process to handle these cases.
  • Adding dynamic queries increases your overall query length and can lead to issues. The server usually has a limit on allowed length for a query. This is defined typically in power of 2, i.e. 4096(2^12) or 8192(2^13) with Successfactors.
  • Process in pages option does not work with Enricher.
  • Leaving the page size empty would be the best for most endpoints. This would ensure all the records are fetched optimally based on server paging configuration.

The above steps might look a bit overwhelming but is relatively easy to perform. The script I shared must suit your use case with minimum changes. In general, dynamic parameters can greatly help improve the performance of your integration scenarios and most adapters support dynamic injection of parameters. Do consider them in all your integration flow designs.

Assigned Tags

      You must be Logged on to comment or reply to a post.
      Author's profile photo Nagarjuna Sreerambhatla
      Nagarjuna Sreerambhatla

      Hi Prasanth Rao,

      Very useful blog , Thank you . One key point I would like to add here is we need to batch/split with group with 999 documents as "in" filter supports only 999 (values) at a time.



      Author's profile photo Prasanth Rao
      Prasanth Rao
      Blog Post Author

      Hello Nagarjuna,

      That's a valid point. I will add this to the "Points to Consider".

      Best regards,


      Author's profile photo Stefan Boller
      Stefan Boller

      Hi Prashanth,


      shouldn't we use the message body as stream instead of converting it to String?


      Best regards



      Author's profile photo Prasanth Rao
      Prasanth Rao
      Blog Post Author

      That's correct. I will correct this. Thank you for pointing it out. 🙂

      Author's profile photo Pushkar Patel
      Pushkar Patel

      Hi Prasanth, Noticed content enricher only supports SF, SOAP and ODATA2 . Is there any r alternative to use it with HTTP or Odata4?

      Author's profile photo Prasanth Rao
      Prasanth Rao
      Blog Post Author

      Presently there are none.

      Author's profile photo Yuvraj Karwar
      Yuvraj Karwar

      Hello  Prasant,


      Thanks for the blog. Its is very useful. But I have one doubt here. In your if there is no 'if' condition directly passed with 'in' filter and keys are blank then what would be output. Is it fetch all records or fetch nothing.There also other other filter with 'and' condition like 'startdate'

      Author's profile photo Prasanth Rao
      Prasanth Rao
      Blog Post Author

      I am afraid I didn't fully understand your question. if the case being discussed is the absence of $filter clause then yes all records will be fetched but it would defeat the purpose of script.

      Regarding additional conditions such as startdate, it needs to be added as required. One can either add it as part of the adapter or even handle it in the script.

      Author's profile photo T S Shankar Narayanan
      T S Shankar Narayanan

      Hi Prashanth,

      my scenario: I get Employee master data using CompoundEmployee SFAPI, and for every employee I have to get additional information from a custom portlet. I am planning to use a Content Enricher to look up SuccessFactors with OData V2, and want to enhance the message with the information from Custom Portlet. So, I will be passing the employee id as the Key Element in the "Original Message" Parameter.

      So, for my scenario, do I still need to pass the $filter parameter with relevant employee ids, as I need the information for all employee id in the XML?

      Author's profile photo Benjamin Nehring
      Benjamin Nehring

      Hello Prasanth,

      thanks for the blog. That's a good approach to only query the data that is really needed. However I see a real problem with the OData URI length limitation of 8192 characters (see KBA 2476062).  So initially this approach might work, but further down into the project the length of external codes of the FOLocation object might be increased and the query might exceed the 8192 length limit.

      BR, Benjamin

      Author's profile photo Daniel Matcovich
      Daniel Matcovich

      Great Blog, do you know if there is any way to use multiple key elements in the join?

      Author's profile photo Urvin Shah
      Urvin Shah

      I am getting request URI error at content enricher