Skip to Content
Technical Articles
Author's profile photo Fellipe Mendes

Cloud Integration – Poll-Enrich with SFTP Adapter for multiple file polling

This blog describes a way to configure the Poll Enrich pattern in combination with the SFTP adapter to transfer files from SAP S/4HANA AL11 (DIR_SAPUSERS) to an external SFTP.

Background Information

While working on a SAP S/4HANA implementation project, there was a requirement to transfer files generated on the SAP S/4HANA AL11 (DIR_SAPUSERS) to an external SFTP.
There are some processes that create these files and that need to be transferred to an external SFTP.

Systems and Platforms

SAP S/4HANA FOUNDATION 2020
SAP BTP Integration Suite -> Cloud Integration -> Cloud Foundry 6.30.17
External SFTP – SSH File Transfer Protocol

Business Requirement

Periodically check the SAP S/4HANA folders for new files. When a file is available, it needs to be transferred to the external SFTP server through Cloud Integration and then the file is moved to the archive folder.

High-level%20process%20overview

High-level process overview

The Challenge with this Integration

With the latest SAP Cloud Integration features, the new Poll Enrich pattern in combination with the SFTP adapter was made available with the main possibilities as mentioned in this blog:

(https://blogs.sap.com/2021/05/12/cloud-integration-use-poll-enrich-with-sftp-adapter/)

– Enrich the message payload with additional data retrieved from a file on the SFTP server
– Poll from a SFTP server triggered by an external trigger, for example triggered via HTTPS call
– Poll from a SFTP server but want to define the configuration in the SFTP adapter dynamically, for example from a partner directory

With this, it is possible to execute a call to an SFTP retrieving a file in the middle of the iFlow, which was previously only possible at the beginning of the iFlow (using the Sender channel).

The challenge begins when Poll-Enrich always polls only one file. If you need to poll multiple files you need to do this Poll-Enrich call in a loop.

However, when you trigger the Poll-Enrich call in a loop, for the first time the file is read (SAP_PollEnrichMessageFound = true); but in the second time, the file is not read  (SAP_PollEnrichMessageFound = false). 

The issue is with the SFTP poll-enrich feature and the post-processing in a loop. 

To avoid this, the recommendation is to implement the loop in one integration flow and from the loop process call another integration flow via ProcessDirect adapter. The second flow then contains the poll-enrich with SFTP, in this way:

Main Flow -> Loop Process Call -> ProcessDirect Adapter -> Second Flow with PollEnrich with SFTP

Solution Implemented 

Two iFLows were created for this solution:

  • First iFlow to orchestrate the pooling
  • Second iFlow to execute the pooling
First IFlow to orchestrate the pooling

First%20IFlow%20to%20orchestrate%20the%20pooling

First IFlow to orchestrate the pooling

  1. The possibility to trigger the integration scenario on-demand by HTTPS call
  2. The possibility to trigger the integration scenario periodically using a Start Timer
  3. The main integration process to:
    • Receive the calls (from block 1 or 2) 
    • Set general properties to control the log level
    • Set the SFTP address by external parameters
    • A Groovy script to get the directories to be accessed from a value mapping:Value%20Mapping%20with%20the%20directories
    • A router to evaluate if has directories or not
    • A value to start the loop
    • Call the process that executes the loop
    • Get the final payload that was combined for each file transferred
    • Log the final payload (execution log) as an attachment in the message (if configured)

      4. A local integration process to transfer all files containing:

    • Get the directory of the current loop stage
    • A router to check if there is directory or not
    • Set the SFTP information like Environment, Folder Source and Folder Destination
    • Define the SFTP environment based on the environment configured
    • Log the beginning of the process 
    • Set the loop condition to true
    • Call the process inside the loop
    • Log the ending of the process

      5. A local integration process to call the next iFlow that pool the file

    • Set the payload that will be transferred to the next iFlow
    • Request Reply to call the next iFlow 
    • Get the results of the next iFLow
    • A router to evaluate if a file was found or not
    • Log the result of process (If a file was found or not)
Second IFlow to execute the pooling

Second%20IFlow%20to%20execute%20the%20pooling

Second IFlow to execute the pooling

  1. The main integration process to:
    • Explore the payload received from the previously iFlow with the SFTP informations
    • A Poll Enrich to fetch the file
    • A SFTP Sender adapter to fetch the file (any file *)
    • A router to evaluate if a file was found or not
    • If found, send to the External SFTP
    • Fill the body with the results of the execution

Execution log

At the end of the process, an execution log will be presented, as following:

Execution%20log

Execution log

Benefits/Advantages of the implemented solution

We can find some benefits and advantages, such as:

    • The file transfer will be automatic, not depending on a person for this
    • An automatic run can be scheduled periodically or also a manual run on demand
    • Multiple directories
    • All files in the directory will be transferred 
    • No need for SAP S/4HANA to communicate directly with the external SFTP
    • The integration scenario is in the Cloud Integration not influencing the functioning of SAP S/4HANA

Possible improvements of the implemented solution

We can find some possible improvements, such as:

    • Transfer files to more than one external SFTP
    • Add exception routines in case of errors

Conclusion

With this solution, it is possible to transfer files from SAP S/4HANA to an external SFTP in a configurable and robust way.

I would be more than happy to hear from our SAP Community about the similar experiences with this requirement.
Please share here in the comments your feedback, thoughts and if you have similar requirements, how you implemented them.

Thank you for reading!!

Fellipe Mendes

Assigned Tags

      9 Comments
      You must be Logged on to comment or reply to a post.
      Author's profile photo Avinash Paul
      Avinash Paul

      Thank you for the Great Blog Fellipe Mendes

      Author's profile photo Fellipe Mendes
      Fellipe Mendes
      Blog Post Author

      Thanks, Avinash

      Author's profile photo Kanan Thiyam
      Kanan Thiyam

      Great blog and is sketched out in a very insightful article. Thanks for sharing.

      Author's profile photo Fellipe Mendes
      Fellipe Mendes
      Blog Post Author

      Thanks, Kanan.

      Author's profile photo Abraham MD
      Abraham MD

      Thanks for the blog. I have one doubt. If S4 is writing the files in AL11 directory then how would CPI pull them? I didn't understand how the solution you designed will read the files from AL11.

       

      Author's profile photo Fellipe Mendes
      Fellipe Mendes
      Blog Post Author

      Hello Abraham.

      Thank you for your question.

      The Basis team provides the connection to the S4 AL11 folder, with this it is possible to configure it in the SAP Cloud Connector and create a virtual host/port for CPI connections.

      We have a timer in CPI that will pull the files periodically.

      Fellipe

       

      Author's profile photo Abraham MD
      Abraham MD

      Hi Fellipe - Thanks for the reply. Could you guide how to connect the AL11 drive to the cloud connector? Any screen grab of the config can help.

      Author's profile photo Fellipe Mendes
      Fellipe Mendes
      Blog Post Author

      Hello Abraham.

      Unfortunately I don't have many details about this configuration since it is a Basis activity and also the purpose of this blog is to present the necessary configurations in the Cloud Integration to process multiples files using the Poll-Enrich with SFTP Adapter.

      I don't know if it helps you but the information I received from the Basis team is that the SFTP is mounted to app servers and added to SAP S/4HANA AL11 (DIR_SAPUSERS).

      The configuration in the SAP Cloud Connector is based on the TCP protocol.

      Hope this can help you in something.

       

      Regards,

      Fellipe

      Author's profile photo Stephan Kallil Teixeira da Silva
      Stephan Kallil Teixeira da Silva

      HI Fellipe. Great blog

      Thanks for the clear explanation