Skip to Content
Author's profile photo Meghna Shishodiya

Cloud Integration – processing SuccessFactor records in batches

This article focuses on how pagination is realized in Cloud Integration for the SuccessFactor adapter.
Please note that batch processing in general is also supported in Cloud Integration for select other connectors. I am writing another article that explains pagination in general. I will link it here once it is published.

Pagination is a feature implemented in SAP Cloud Platform Integration to be able to process large requests in pages.

When pagination is configured, the sender SFSF system sends only one page to Cloud Integration at a time. When Cloud Integration finishes processing the page, it requests for the next page. You can choose to collate all the pages later before sending out the final response or send the response page by page to the receiver.

As of today, pagination is only supported for SFSF SOAP and OData V2 communication.

In order to enable pagination in your scenario, you need to make the following settings in your integration flow:

1)      Move all the logic of processing the message into a sub-process.

2)      Invoke the sub-process from the main process using a looped process call:

 

 

The condition expression refers to the name of the SFSF channel that polls request from the SFSF system.

${property.SAP_SuccessFactorsHasMoreRecords.QueryCompoundEmployee.<Name of the channel polling SFSF data>} contains ‘true’

Set the value for ‘Max. Number of Iterations’ appropriately to ensure that it is not:

  1. Too high – if you know that typical number of expected pages
  2. Too low – that it leads to truncation of incoming pages – it should be able to capture all the data and not get truncated due to a cap on the number of maximum iterations

The condition expression, together with the Max number of iteration are meant to save the system from going into an endless loop.

3)      The SFSF channel should be configured inside the sub-process using the request-reply pattern.

4)      Configure the following settings in the SFSF  channel:

  • Define a name for the channel that you can use for defining the condition expression in step 2
  • Define a page size keeping in mind:
    • It should be able to capture all the data and not get truncated due to a cap on the number of maximum iterations
    • It should not too big that we encounter OOMs.
  • Check the checkbox “Process in batches”

If you do not see these options, delete the channel and create a fresh one.

5)      Finally, any housekeeping or post-processing tasks should be put in the main process after the call to the looping process.

Assigned tags

      10 Comments
      You must be Logged on to comment or reply to a post.
      Author's profile photo sandeep palicherla
      sandeep palicherla

      good blog,keep posting.please share step by step process.

      Thank you...

      Author's profile photo Former Member
      Former Member

      When adding the  Process Call Condition as mentioned here, I am geting the error: Invalid format of condition expression value.

      Author's profile photo Meghna Shishodiya
      Meghna Shishodiya
      Blog Post Author

      Hello Markus, Can you send me the expression you are using. Thank you!

      Author's profile photo Former Member
      Former Member

      It was finally a copy/paste issue. Quotes were ` instead of '. Thanks anyways!

      Author's profile photo Meghna Shishodiya
      Meghna Shishodiya
      Blog Post Author

      Hello Markus, can you please share the expression you are using. Thank you!

      Author's profile photo Debashis Mohanty
      Debashis Mohanty

      Hello Meghna,

      Is there a way to store/log/capture the number of times the Looping Process Call might have run ?

      Regards,

      Debashis

      Author's profile photo Meghna Shishodiya
      Meghna Shishodiya
      Blog Post Author

      Hello Debashis, Yes, you can define a property in the main process and increment it every time you enter the looped process. At the end of the loop, you can check the value of the variable to know the number of runs.

      Thanks! Meghna

      Author's profile photo Sven Burow
      Sven Burow

      Hi Debashis, SAP CPI creates a property which counts the number of runs: CamelLoopIndex

      Author's profile photo Bence Somlyo
      Bence Somlyo

      Hello Meghna,

      if I copy and paste the expression:

      ${property.SAP_SuccessFactorsHasMoreRecords.QueryCompoundEmployee.<Name of the channel polling SFSF data>} contains ‘true’

      then the CPI fails with the following exception as above: “Invalid format of condition expression value.”

      If we check the expression closer, we will find two strange characters being used instead of simple quotes around the true . Could you change the expression in the blog? If someone uses copy and paste option, it takes time to troubleshoot this, because the characters are very similar. It is better to use this one:

      ${property.SAP_SuccessFactorsHasMoreRecords.QueryCompoundEmployee.<Name of the channel polling SFSF data>} contains 'true'

      Best regards,
      Bence

      Author's profile photo Arnab Maji
      Arnab Maji

      Hello,

       

      Please use the format as mentioned below.

       

      ${property.SAP_SuccessFactorsHasMoreRecords.<Reciever>.<Channel>} = 'true'

       

       

      Regards,

      Arnab Maji