Handling Large Data With SAP Cloud Platform Integration OData V2 Adapter

SAP Cloud Platform Integration helps you achieve integration with OData services. One of the key elements for integration is processing and exchange of large amounts of data. To ensure that these scenarios are executed without any issues, especially related to memory, you can use the Process in Pages option provided by the OData V2 adapter/SuccessFactors OData V2 adapter in SAP Cloud Platform Integration. You need to use this adapter in a Local Integration Process and invoke that process using a Looping Process Call to use the Process in Pages option.

This option enables the adapter to fetch only a limited number of records in one message processing cycle. This number is defined by the value that you provide in the Page Size field of the adapter configuration.

1: You must specify the Page Size based on the number of records that you want the system to fetch in one page of results.

2: This option enables you to fetch records in pages of size defined in Page Size. For example, if you have 100 records and you set the Page Size as ’10’, the adapter fetches data in pages containing 10 records/page.

Important Consideration for Using Process in Pages

To use this option, you must ensure that you are using the OData adapter in a Local Integration Process that is invoked by a Looping Process Call. The looping process triggers the message processing multiple times depending on the configuration and ensures that all the pages are retrieved from the receiver system. For example, let us assume that there are 100 records that you want to fetch, and each page should contain 10 records. You configure the Page Size in OData receiver adapter as ’10’ and configure the Looping Process Call to run a minimum of 10 iterations to ensure that all the records are fetched.

Advantage of Using this Approach

It is recommended that you use the Process in Pages with Looping Process Call option while handling heavy loads of data. By doing so, you ensure that the memory consumption is optimized and the scenarios run without any issues. The overall load on the tenant will also be moderate by using this approach while working with large amount of data.

Example Integration Flow

Here’s a  scenario (using OData V2 adapter) where you can understand how to use this approach to fetch large amounts of data from a OData service. Please note that this screenshot is being used for indicative purpose only. You can add other flow steps after the Looping Process Call based on your requirement and complete the modeling of the integration flow. You can also use a sender system with an adapter in the sender channel if required.

This integration flow is designed to fetch all the records from OData service in pages containing a configured number of records. The Looping Process Call will invoke the Local Integration Process, which will fetch one page of record from the OData service and send it to Receiver 2 through mail adapter. This process is executed repeatedly till the Condition Expression is satisfied or Maximum No. of Iterations is completed.

After you deploy the integration flow, only the Looping Process Call step is executed repeatedly. Please note that the integration process will only be executed once and you see only one message in the Message Monitoring tab. For each iteration of the Looping Process Call, one page of records is fetched from the OData service.

If you want to manipulate each page of data (mapping, convert into another format, push data to target system, etc.), you must add these integration flow steps inside the Local Integration Process. The Integration Process (main process) will get only the last page of the records fetched.

You can choose and configure the adapters in the Integration Process and Local Integration Process based on your requirement. In the OData receiver adapter that you have used in Local Integration Process, ensure that you select Process in Pages option. In the Page Size field, provide the value for the number of records that you want in one message.

Here are the values that you can use in the Looping Process Call after you assign the Local Integration Process 1 to it using Select:

Expression Type: Non-XML

Condition Expression:

${property.Receiver1.OData.hasMoreRecords} contains 'true' 

Max. Numbers of Iterations: <total number of iterations required to fetch all records>

The syntax of condition expression is

${property.<receivername>.<channelname>.hasMoreRecords} contains 'true'

For more information, refer to OData adapter documentation.

Significance of Max. Number of Iterations

1. By default, the value of Max. Numbers of Iterations is set to 100. If you wish to manually provide this, you can calculate this value by dividing total number of records by Page Size.
2. It is mandatory to have a value for Max. Numbers of Iterations to ensure that the integration flow is not stuck in an endless loop in case of any issues with the looping.
3. If you are unsure about the maximum iterations value, enter the value as 99999 (upper limit).
4. If Condition Expression evaluation fails, loop will be only executed once. Ensure values for <receivername>, <channelname> etc are provided accurately.

Additional Information About This Scenario

What data is returned to the main integration process from local integration process in this integration flow?

When you deploy this integration flow, the Local Integration Process will send only the last page of the records fetched from the OData service to the Integration Process.

For example, consider this scenario:

Here, a receiver system is accessed via Mail adapter (Mail 2) in Integration Process. If you configure the adapter to send an email to your email ID, you will get the last page of records fetched from the OData service. If you consider a service that contains 75 records, and you have specified the Page Size as 10, you will receive eight emails from Mail 1 adapter and one email having 5 records from Mail 2.

Please note that this is under the assumption that no further records are added to the OData service when the data is being fetched from it by the OData adapter. 

What happens when pagination fails?

If there is a failure in processing a page of data (communication error in fetching the 4th page out of 10, or any flow step within the Local Integration Process throws an error, issue in the adapter configuration of Mail 1, etc.), the entire message processing will fail.

How to handle exceptions?

If you want to handle the exceptions which might occur in processing the data in pages inside Local Integration Process, you can use an Exception Subprocess. In the example below, if there is any error in the Local Integration Process, a message will be sent to the Receiver via Mail adapter.

Related Links

OData Adapter Configuration

SuccessFactors (OData V2) Adapter Configuration

Looping Process Call

User Assistance Overview Product Page for SAP Cloud Platform Integration