Cloud Integration – processing SuccessFactor records in batches
This article focuses on how pagination is realized in Cloud Integration for the SuccessFactor adapter.
Please note that batch processing in general is also supported in Cloud Integration for select other connectors. I am writing another article that explains pagination in general. I will link it here once it is published.
Pagination is a feature implemented in SAP Cloud Platform Integration to be able to process large requests in pages.
When pagination is configured, the sender SFSF system sends only one page to Cloud Integration at a time. When Cloud Integration finishes processing the page, it requests for the next page. You can choose to collate all the pages later before sending out the final response or send the response page by page to the receiver.
As of today, pagination is only supported for SFSF SOAP and OData V2 communication.
In order to enable pagination in your scenario, you need to make the following settings in your integration flow:
1) Move all the logic of processing the message into a sub-process.
2) Invoke the sub-process from the main process using a looped process call:
The condition expression refers to the name of the SFSF channel that polls request from the SFSF system.
${property.SAP_SuccessFactorsHasMoreRecords.QueryCompoundEmployee.<Name of the channel polling SFSF data>} contains ‘true’
Set the value for ‘Max. Number of Iterations’ appropriately to ensure that it is not:
- Too high – if you know that typical number of expected pages
- Too low – that it leads to truncation of incoming pages – it should be able to capture all the data and not get truncated due to a cap on the number of maximum iterations
The condition expression, together with the Max number of iteration are meant to save the system from going into an endless loop.
3) The SFSF channel should be configured inside the sub-process using the request-reply pattern.
4) Configure the following settings in the SFSF channel:
- Define a name for the channel that you can use for defining the condition expression in step 2
- Define a page size keeping in mind:
- It should be able to capture all the data and not get truncated due to a cap on the number of maximum iterations
- It should not too big that we encounter OOMs.
- Check the checkbox “Process in batches”
If you do not see these options, delete the channel and create a fresh one.
5) Finally, any housekeeping or post-processing tasks should be put in the main process after the call to the looping process.
good blog,keep posting.please share step by step process.
Thank you...
When adding the Process Call Condition as mentioned here, I am geting the error: Invalid format of condition expression value.
Hello Markus, Can you send me the expression you are using. Thank you!
It was finally a copy/paste issue. Quotes were ` instead of '. Thanks anyways!
Hello Markus, can you please share the expression you are using. Thank you!
Hello Meghna,
Is there a way to store/log/capture the number of times the Looping Process Call might have run ?
Regards,
Debashis
Hello Debashis, Yes, you can define a property in the main process and increment it every time you enter the looped process. At the end of the loop, you can check the value of the variable to know the number of runs.
Thanks! Meghna
Hi Debashis, SAP CPI creates a property which counts the number of runs: CamelLoopIndex
Hello Meghna,
if I copy and paste the expression:
${property.SAP_SuccessFactorsHasMoreRecords.QueryCompoundEmployee.<Name of the channel polling SFSF data>} contains ‘true’
then the CPI fails with the following exception as above: “Invalid format of condition expression value.”
If we check the expression closer, we will find two strange characters being used instead of simple quotes around the true . Could you change the expression in the blog? If someone uses copy and paste option, it takes time to troubleshoot this, because the characters are very similar. It is better to use this one:
${property.SAP_SuccessFactorsHasMoreRecords.QueryCompoundEmployee.<Name of the channel polling SFSF data>} contains 'true'
Best regards,
Bence
Hello,
Please use the format as mentioned below.
${property.SAP_SuccessFactorsHasMoreRecords.<Reciever>.<Channel>} = 'true'
Regards,
Arnab Maji
Hi, at last version in CPI the iflow has configured and waiting for SAP_SuccessFactorsHasMoreRecords value 'TRUE' but from SFSF comes 'true' and it's getting error. Please, could someone help us?