Skip to Content
Author's profile photo Meghna Shishodiya

Cloud Integration – processing SuccessFactor records in batches

This article focuses on how pagination is realized in Cloud Integration for the SuccessFactor adapter.
Please note that batch processing in general is also supported in Cloud Integration for select other connectors. I am writing another article that explains pagination in general. I will link it here once it is published.

Pagination is a feature implemented in SAP Cloud Platform Integration to be able to process large requests in pages.

When pagination is configured, the sender SFSF system sends only one page to Cloud Integration at a time. When Cloud Integration finishes processing the page, it requests for the next page. You can choose to collate all the pages later before sending out the final response or send the response page by page to the receiver.

As of today, pagination is only supported for SFSF SOAP and OData V2 communication.

In order to enable pagination in your scenario, you need to make the following settings in your integration flow:

1)      Move all the logic of processing the message into a sub-process.

2)      Invoke the sub-process from the main process using a looped process call:

 

 

The condition expression refers to the name of the SFSF channel that polls request from the SFSF system.

${property.SAP_SuccessFactorsHasMoreRecords.QueryCompoundEmployee.<Name of the channel polling SFSF data>} contains ‘true’

Set the value for ‘Max. Number of Iterations’ appropriately to ensure that it is not:

  1. Too high – if you know that typical number of expected pages
  2. Too low – that it leads to truncation of incoming pages – it should be able to capture all the data and not get truncated due to a cap on the number of maximum iterations

The condition expression, together with the Max number of iteration are meant to save the system from going into an endless loop.

3)      The SFSF channel should be configured inside the sub-process using the request-reply pattern.

4)      Configure the following settings in the SFSF  channel:

  • Define a name for the channel that you can use for defining the condition expression in step 2
  • Define a page size keeping in mind:
    • It should be able to capture all the data and not get truncated due to a cap on the number of maximum iterations
    • It should not too big that we encounter OOMs.
  • Check the checkbox “Process in batches”

If you do not see these options, delete the channel and create a fresh one.

5)      Finally, any housekeeping or post-processing tasks should be put in the main process after the call to the looping process.

Assigned Tags

      12 Comments
      You must be Logged on to comment or reply to a post.
      Author's profile photo sandeep palicherla
      sandeep palicherla

      good blog,keep posting.please share step by step process.

      Thank you...

      Author's profile photo Former Member
      Former Member

      When adding the  Process Call Condition as mentioned here, I am geting the error: Invalid format of condition expression value.

      Author's profile photo Meghna Shishodiya
      Meghna Shishodiya
      Blog Post Author

      Hello Markus, Can you send me the expression you are using. Thank you!

      Author's profile photo Former Member
      Former Member

      It was finally a copy/paste issue. Quotes were ` instead of '. Thanks anyways!

      Author's profile photo Meghna Shishodiya
      Meghna Shishodiya
      Blog Post Author

      Hello Markus, can you please share the expression you are using. Thank you!

      Author's profile photo Debashis Mohanty
      Debashis Mohanty

      Hello Meghna,

      Is there a way to store/log/capture the number of times the Looping Process Call might have run ?

      Regards,

      Debashis

      Author's profile photo Meghna Shishodiya
      Meghna Shishodiya
      Blog Post Author

      Hello Debashis, Yes, you can define a property in the main process and increment it every time you enter the looped process. At the end of the loop, you can check the value of the variable to know the number of runs.

      Thanks! Meghna

      Author's profile photo Sven Burow
      Sven Burow

      Hi Debashis, SAP CPI creates a property which counts the number of runs: CamelLoopIndex

      Author's profile photo Bence Somlyo
      Bence Somlyo

      Hello Meghna,

      if I copy and paste the expression:

      ${property.SAP_SuccessFactorsHasMoreRecords.QueryCompoundEmployee.<Name of the channel polling SFSF data>} contains ‘true’

      then the CPI fails with the following exception as above: “Invalid format of condition expression value.”

      If we check the expression closer, we will find two strange characters being used instead of simple quotes around the true . Could you change the expression in the blog? If someone uses copy and paste option, it takes time to troubleshoot this, because the characters are very similar. It is better to use this one:

      ${property.SAP_SuccessFactorsHasMoreRecords.QueryCompoundEmployee.<Name of the channel polling SFSF data>} contains 'true'

      Best regards,
      Bence

      Author's profile photo Arnab Maji
      Arnab Maji

      Hello,

       

      Please use the format as mentioned below.

       

      ${property.SAP_SuccessFactorsHasMoreRecords.<Reciever>.<Channel>} = 'true'

       

       

      Regards,

      Arnab Maji

      Author's profile photo ANA MARTINEZ RAMIREZ
      ANA MARTINEZ RAMIREZ

      Hi, at last version in CPI the iflow has configured and waiting for SAP_SuccessFactorsHasMoreRecords value 'TRUE' but from SFSF comes 'true' and it's getting error. Please, could someone help us?

      Author's profile photo suryavamsi kopparapu
      suryavamsi kopparapu

      Hi Meghana,

       

      Thanks for nice blog. But if i use this process the LSRD (execution timestamp from API is changing). if my records are 50000 employees then i have observed more than 35 mins(means execution timestamp for 1st run is 01:01 min and execution timestamp for last run is 01:35). So,

      Case1:

      if any employee is processed which is changed at 01:05 and it is picked up in 01:30 run. in this case if i use LSRD as 01:01 then that employee is again picked.

      Case2:

      viceversa of same employee: if employee is processed at 01:01 1st loop and that employee is changed at 01:23 and if i use LSRD as 01:35 then that employee will be never picked up.

       

      35 mins integration time is very less, many payroll/benifits integrations are taking more than 2 hrs for processiong.

      Please tell how can i handle this process.

       

      Thanks & Regards,

      Suryavamsi Kopparapu