Skip to Content
Technical Articles
Author's profile photo Gautham Krishna

Handling Large Data With SAP Cloud Platform Integration OData V2 Adapter

SAP Cloud Platform Integration helps you achieve integration with OData services. One of the key elements for integration is processing and exchange of large amounts of data. To ensure that these scenarios are executed without any issues, especially related to memory, you can use the Process in Pages option provided by the OData V2 adapter/SuccessFactors OData V2 adapter in SAP Cloud Platform Integration. You need to use this adapter in a Local Integration Process and invoke that process using a Looping Process Call to use the Process in Pages option.

This option enables the adapter to fetch only a limited number of records in one message processing cycle. This number is defined by the value that you provide in the Page Size field of the adapter configuration.

1: You must specify the Page Size based on the number of records that you want the system to fetch in one page of results.

2: This option enables you to fetch records in pages of size defined in Page Size. For example, if you have 100 records and you set the Page Size as ’10’, the adapter fetches data in pages containing 10 records/page.

Important Consideration for Using Process in Pages

To use this option, you must ensure that you are using the OData adapter in a Local Integration Process that is invoked by a Looping Process Call. The looping process triggers the message processing multiple times depending on the configuration and ensures that all the pages are retrieved from the receiver system. For example, let us assume that there are 100 records that you want to fetch, and each page should contain 10 records. You configure the Page Size in OData receiver adapter as ’10’ and configure the Looping Process Call to run a minimum of 10 iterations to ensure that all the records are fetched.

Advantage of Using this Approach

It is recommended that you use the Process in Pages with Looping Process Call option while handling heavy loads of data. By doing so, you ensure that the memory consumption is optimized and the scenarios run without any issues. The overall load on the tenant will also be moderate by using this approach while working with large amount of data.

Example Integration Flow

Here’s a  scenario (using OData V2 adapter) where you can understand how to use this approach to fetch large amounts of data from a OData service. Please note that this screenshot is being used for indicative purpose only. You can add other flow steps after the Looping Process Call based on your requirement and complete the modeling of the integration flow. You can also use a sender system with an adapter in the sender channel if required.

This integration flow is designed to fetch all the records from OData service in pages containing a configured number of records. The Looping Process Call will invoke the Local Integration Process, which will fetch one page of record from the OData service and send it to Receiver 2 through mail adapter. This process is executed repeatedly till the Condition Expression is satisfied or Maximum No. of Iterations is completed.

After you deploy the integration flow, only the Looping Process Call step is executed repeatedly. Please note that the integration process will only be executed once and you see only one message in the Message Monitoring tab. For each iteration of the Looping Process Call, one page of records is fetched from the OData service.

If you want to manipulate each page of data (mapping, convert into another format, push data to target system, etc.), you must add these integration flow steps inside the Local Integration Process. The Integration Process (main process) will get only the last page of the records fetched.

You can choose and configure the adapters in the Integration Process and Local Integration Process based on your requirement. In the OData receiver adapter that you have used in Local Integration Process, ensure that you select Process in Pages option. In the Page Size field, provide the value for the number of records that you want in one message.

Here are the values that you can use in the Looping Process Call after you assign the Local Integration Process 1 to it using Select:

Expression Type: Non-XML

Condition Expression:

${property.Receiver1.OData.hasMoreRecords} contains 'true' 

Max. Numbers of Iterations: <total number of iterations required to fetch all records>

The syntax of condition expression is

${property.<receivername>.<channelname>.hasMoreRecords} contains 'true'

For more information, refer to OData adapter documentation.

Significance of Max. Number of Iterations

  1. By default, the value of Max. Numbers of Iterations is set to 100. If you wish to manually provide this, you can calculate this value by dividing total number of records by Page Size.
  2. It is mandatory to have a value for Max. Numbers of Iterations to ensure that the integration flow is not stuck in an endless loop in case of any issues with the looping.
  3. If you are unsure about the maximum iterations value, enter the value as 99999 (upper limit).
  4. If Condition Expression evaluation fails, loop will be only executed once. Ensure values for <receivername>, <channelname> etc are provided accurately.

Additional Information About This Scenario

What data is returned to the main integration process from local integration process in this integration flow?

When you deploy this integration flow, the Local Integration Process will send only the last page of the records fetched from the OData service to the Integration Process.

For example, consider this scenario:

Here, a receiver system is accessed via Mail adapter (Mail 2) in Integration Process. If you configure the adapter to send an email to your email ID, you will get the last page of records fetched from the OData service. If you consider a service that contains 75 records, and you have specified the Page Size as 10, you will receive eight emails from Mail 1 adapter and one email having 5 records from Mail 2.

Please note that this is under the assumption that no further records are added to the OData service when the data is being fetched from it by the OData adapter. 

What happens when pagination fails?

If there is a failure in processing a page of data (communication error in fetching the 4th page out of 10, or any flow step within the Local Integration Process throws an error, issue in the adapter configuration of Mail 1, etc.), the entire message processing will fail.

How to handle exceptions?

If you want to handle the exceptions which might occur in processing the data in pages inside Local Integration Process, you can use an Exception Subprocess. In the example below, if there is any error in the Local Integration Process, a message will be sent to the Receiver via Mail adapter.

OData Adapter Configuration

SuccessFactors (OData V2) Adapter Configuration

Looping Process Call

User Assistance Overview Product Page for SAP Cloud Platform Integration

Assigned Tags

      You must be Logged on to comment or reply to a post.
      Author's profile photo Nick Yang
      Nick Yang

      Hi Gautham,


      Thanks for this great blog. But, don't know why all the pictures in the blog are unable to display.

      Would you be able to check it? Thanks.





      Author's profile photo Gautham Krishna
      Gautham Krishna
      Blog Post Author

      Hi Nick,

      Thank you for your feedback.

      Can you please check in Google Chrome? I checked and the images are being displayed properly.

      Thanks & Warm Regards,


      Author's profile photo Melissa Sullivan
      Melissa Sullivan


      We are not able to view the embedded images.  We have tried in both Chrome and IE.  Do you have any other suggestions?


      Author's profile photo Gautham Krishna
      Gautham Krishna
      Blog Post Author

      Hello Melissa,

      Thank you for getting in touch. There was an issue with the uploaded images which has been fixed now. If you are still facing the same issue, please let me know.

      Thanks & Regards, Gautham

      Author's profile photo Suresh Sakthivel
      Suresh Sakthivel

      Hi Gautham Krishna ,

      Thanks for the Nice blog!!

      we have requirement to send one file once all iteration is over.

      We need to collate the data and send it to SFTP File server as a single file.

      Would it be possible, or we would be transferring multiple files only.

      Best Regards,
      Suresh S

      Author's profile photo Gautham Krishna
      Gautham Krishna
      Blog Post Author

      Hi Suresh,

      In this scenario, you cannot collate the data. You have to transfer multiple files.

      Best Regards,


      Author's profile photo Maik Offerle
      Maik Offerle

      Hi Suresh Sakthivel

      An alternative solution would be to collect all the data in the looping process call in a data store and this data store is then handled in the integration process



      Author's profile photo Axel Albrecht
      Axel Albrecht

      Hi Suresh,

      We do support the “append” mode for SFTP receivers by which you can extend the file with every page. By this you will end up with 1 file instead of multiple files.


      Author's profile photo Jonas Wang
      Jonas Wang

      Hi ,

      Is it possible to implement this solution using content-enricher instead of request-reply?

      Author's profile photo Pushkar Patel
      Pushkar Patel

      Hi Gautham,

      Thanks for this informative blog.

      Whats the maximum file size recommendation for SFTP Sender in CPI ? Whats the best way if I want to process a big file upto 50-60 MB which involves mapping and transformation.




      Author's profile photo Zain Seedat
      Zain Seedat

      Thank you for the blog.


      Since the Local Integration Process will send only the last page of the records fetched from the OData service to the Integration Process.


      What are the options to retrieve all data in a single payload outside of the loop?

      I have tried to use a message store - but this overwrites each time the loop is processed.

      Kind Regards

      Author's profile photo Dinesh M
      Dinesh M

      Hello Gautam,

      I have tried this and it is working like a charm.

      I understood if I give pagesize as 2 and if I have 5 records there will be 3 iterations run by the system but if I go deep to understand what that property is , I'm getting confused as that property is not present in the standard header list as well

      Could you brief a bit on this below line how system understands how many records are fetched by the receiver and splits the iteration accordingly

      ${property.Receiver1.OData.hasMoreRecords} contains 'true' 





      Author's profile photo venkata sai ch
      venkata sai ch

      NIce blog thanks