Technical Articles
Cloud Integration – Use Poll-Enrich with SFTP Adapter
This blog describes how to configure the new Poll Enrich pattern in combination with the sftp adapter, elaborating on use cases and different configuration options.
Use Poll Enrich with SFTP Adapter
In general, there are three main use cases for the Poll Enrich pattern:
- You want to enrich the message payload with additional data retrieved from a file on the sftp server.
- You want to poll from an sftp server triggered by an external trigger, for example triggered via HTTP call.
- You need to poll from an sftp server but want to define the configuration in the sftp adapter dynamically, for example from a partner directory.
In this blog, I will focus on the second and third use case. Yet, I will shortly also mention the configuration option for the first use case when discussing the integration flow configuration.
This is the simple integration flow we are going to build:
The flow starts with a HTTP inbound adapter that triggers the message processing. In a subsequent Content Modifier, some properties are defined that are used dynamically in the sftp adapter when polling the file from sftp server. At the very end, the message is sent out to the receiver system.
Let’s start.
In the integration flow, draw a connection from the Sender participant to the Start Message event and select the HTTP adapter; then add a Content Modifier next to it. Configure the HTTP adapter as you like.
In the Content Modifier, you define the properties that are to be used dynamically in the SFTP Adapter. For dynamically configurable settings, refer to the blog Cloud Integration – Dynamically Configure the SFTP Receiver Adapter. The same settings are allowed dynamically in the sftp adapter in the Poll-Enrich pattern. Note that with the first increment (version 1.0 of the Poll Enrich sftp adapter), Proxy Type and Authentication can’t yet be set dynamically.
For my simple scenario, I define the following properties to set them dynamically via XPath from the incoming http request:
Next to the Content Modifier, configure the Poll Enrich step. For this, in the Call section of the modeler, select External Call and then Poll Enrich.
In the Poll Enrich step, we have to configure how the payloads need to be aggregated: the one before the Poll Enrich step and the payload fetched from sftp server need. In addition you configure what to do if no message is found on the sftp server.
The aggregation is configured with the help of the Aggregation Algorithm in the Poll Enrich step:
For our use case (to simply read files via external trigger or via dynamic configuration in the sftp adapter), the Replace option is the correct choice. This algorithm simply replaces the original message (payload and headers) before the Poll Enrich step with the new one fetched from the sftp server.
If you want to enrich your original payload (use case 1), you have to select one of the other two algorithms:
- Combine XML: This option combines the message polled from the sftp server with the original message in multimapping format. This option makes most sense if both, the original and the polled message, are XML messages.
- Concatenate: The message polled from the sftp server is added to the existing message. This option is useful for strings.
If you want to get an exception in case no message is found on the sftp server you can select the checkbox Throw Exception if no Message Found. If not selected, the header SAP_PollEnrichMessageFound is set to false if no message can be found during processing. Based on the header you can configure alternative processing.
Having defined the Poll Enrich step, we now need to configure the SFTP adapter. To do so, add another Sender participant and draw a connection from this new Sender participant to the Poll Enrich step. Select the SFTP adapter.
The configuration of the SFTP adapter will look familiar to you because it contains almost the same configuration options as the normal SFTP sender adapter. Some configurations, like the Scheduler tab, are missing because they are not relevant in this pattern, but most settings are equal.
In the configuration we specify the properties defined in the Content Modifier:
As a final step, configure a connection to the Receiver participant to send the message out.
Deploy the flow and trigger it with a payload that provides the necessary data; in my example, I use the following simple payload:
<?xml version='1.0' encoding='UTF-8' ?>
<request>
<sftp>
<Directory>test/test1</Directory>
<File>TestFile*</File>
</sftp>
</request>
Note that in case no file can be found on the sftp server, the processing continues with the payload before the Poll Enrich step. It is planned to offer specific handling in such a case.
I hope this simple scenario helps you to understand the Poll Enrich pattern.
Planned Extensions:
- Dynamic Configuration of Proxy Type and Authentication (available after the June 2021 update)
- Keep File and Process Again option not available (available with July 2021 update)
- Configure Handling if File is not found on the sftp server (available with August 2021 update)
- Support FTP adapter in Poll Enrich step
Hi Mandy,
this is a dream feature we waited from 4 years, thank you very much, could you ask to your developer to add on Process the option None, a lot of time we just need to read files without variation of timestamp and position.
We tried some workaround like take file move on processing folder and at end on workflow put it bak on sftp original folder.
Destination folder doesn't accept properties, headers like parameter and if you put . instead .archive it doesn't work, files disappears on void :).
Thank you and good job!
Hello Federico,
thank you for your nice feedback.
Concerning option None you mean to simply keep the file and process it again with the next poll? This will come with the next update, it will be called 'Keep File and Process Again'. Note that you need to recreate the sftp channel to get the new version of the polling sftp sender adapter to get this feature.
Best regards,
Mandy
As you can see there is no way to keep file for next run....
As said, this feature will come with the next update only. This is in about 2-3 weeks. Afterwards you will find it 🙂
Best regards
Mandy
Hi Mandy Krimmel ,
With Poll Enrich, when I am trying to use SFTP adapter and post processing as delete, its not working. Is there any limitation around deletion the file after processing with Poll Enrich ?
Thanks,
Pushkar
This should work, maybe there is a problem with permission to delete the file? Maybe check the log files in the sftp server.
Best regards
Mandy
Hi Mandy,
Thank you for sharing this information, it's really efforts saving functionality in middleware. In omy current development we have FTP server ,any tentative date when we are planning to have this functionality for FTP?
Thank you,
Monica
It is planned for the September or October update, depending how well the testing goes.
Best regards,
Mandy
Thank you for quick reply and update Mandy, it's really helpful .Just check roadmap plan and it's in Q3, eagerly waiting for this feature as this will help to reduce lots of customize efforts in development .
Hi Mandy, In this SFTP poll enrich processing option for sorting and polling is missing, when will this be available. Or is this by design where you have to pass the dir/filename as a variable to poll enrich > ftp?
Thanks
This is as per design, the sorting does not work together with the poll-enrich step. Because of this this option is not available. If you need a specific file you need to pass the file name as header/property.
BR
Mandy
Hi Mandy,
Very useful blog.
I just want to know the differences between Content enricher via SFTP and Poll enricher. I understand that poll enricher has better aggregation strategies.
Want to know the other differences reason being poll enricher works for sender and Content enricher works with a receiver. However both of these are actually polling SFTP based on the file name, directory details we configure. They poll the file data and enrich as per aggregation strategy.
Regards,
Santosh
Hello Santosh,
Content Enricher is always using the response coming back from the receiver to aggregate. It is using a receiver channel. And if you choose a receiver channel you can write files to sftp adapter but not poll files from there. The polling of files in the middle of the flow only works with the poll enricher as this is using a sftp sender channel.
Content Enricher is usually used with HTTP based adapters that send a response and this response is to be aggregated to the payload.
Best regards
Mandy
Hi Mandy,
Thanks for this. We wanted to check whether the Sender can also be an SFTP? We wanted to be able to process and incoming SFTP file and perform a lookup on another SFTP file using the poll enricher.
Thanks.
Regards,
Madina
Yes, this is possible.
BR
Mandy
Thanks, Mandy! Indeed the CPI Consultant confirmed this.
Cheers,
Madina
Hi Mandy,
Poll enrich will poll all the files and combine the all files data which are available in Sftp with original message or only single file data will combine with original message?
Thanks,
It will poll only one file at one execution.
BR
Mandy
Hi Mandy,
so if it only polls one file from a directory (even if there are more files) and my requirement is, to leave all files untouched after reading, how can I achieve to read all the other files?
Regards
Stefan
Hello Stefan,
if you want to not touch the file in the sftp server you may use the post-processing option to mark the file in Idempotent Repository. With this the file is not polled anymore even if not moved or deleted on the sftp server.
Or do I not understand the scenario correctly and you even want to poll the files again in a subsequent processing?
Regards
Mandy
Hi Mandy,
thank you for your reply. With the post-processing option it should work for me.
The files on the sftp server won't be deleted, but every week they will be replaced, meaning, the names of the files won't change, only their file-data. But I assume once marked and then replaced I can mark the files the following week again?
Regards
Stefan
Hi Mandy,
I have to pick files from the SFTP folder in the format ABC20211207.txt
I have tried to give regular expression as - ABC[0-9]{8}\.txt, ABC*.txt, ABC.*\.txt. But the file is not getting picked no matter what regular expression I give. Can you please let me know what can I give in the File name to pick the files in the above format.
Appreciate your help,
Sunitha Mamidi
Hello,
please refer to Configure the SFTP Sender Adapter Used with the Poll Enrich Step - SAP Help Portal.
ABC*.txt should work, the others not.
Best regards
Mandy
Hi Mandy,
Thank you for your quick response.
If I give this regular expression (ABC*.txt), The file is getting picked.
But when I select the option in processing tab, Read Lock Strategy --> Done File Expected (As per our requirement). The file is not getting picked. What is causing the issue?
Thank you,
Sunitha
Hi Sunitha,
how did you define the done file?
Best regards
Mandy
Hi Mandy,
Thank you for your time.
We have 2 IFlows for this scenario -
The .done file is available in the SFTP folder for my scenario. If I select this option and give the regular expression as above, The file is not getting picked.
Regards,
Sunitha
Hello Sunitha,
we currently have an issue with the dynamic done file setting. We are working on a fix for this.
Could you please try the following as a workaround:
Please let my know if this workaround solves your issue.
Best regards
Mandy
Hi Mandy,
The workaround solved my issue.
Thank you so much for your valuable time.
thank you for the info. Glad it solved your issue.
Hi Mandy,
thanks for your useful blogpost!
How can I get the filename from the sftp server? Neither the sftp nor the poll enrich adapter provides such a property or am I missing a detail?
BR, Gunnar
CamelFileName is the header that is set be the adapter to give the filename. Please check this in trace. You may have to set the header in the Allowed Headers Configuration in the integration flow configuration.
Best regards
Mandy
Hi Mandy,
Thanks for this blog 🙂 can you please help me on my current scenario that after reading the file i need to catch the file content for encode this to base64 ?. Is it this possible to do it ?.
Can you please help me with an example if you have ?.
Thanks
Best regards.
I dont fully understand your requirement. If the file is polled via poll-enrich the file content is available in the exchange and can be processed for example via the base64 encoder to convert the payload to base64 format.
Best regards
Mandy
Hi Mandy,
Great article. But I need to read all the files in the SFTP folder. I am trying to use pollEnrich with SFTP sender with CombinedXML option. It seems like not working. I am seeing only one file content in the log. Please let me know if you have any suggestions.
Best Regards
Ramana Reddy
Poll-Enrich always only fetches one file. If you want to fetch multiple files you need to call the sftp in Poll-Enrich in a loop.
Best regards
Mandy
Thank you Mandy Krimmel. Is there any approach to get the count of the files from SFTP to loop those many number of times to get all the files. Sorry I am new to CPI.
Best Regards
Ramana Reddy
No, unfortunately there is no option to get the count. It would probably be best to loop as long as there are files returned and stop if no file is retrieved anymore.
Please note, currently there is an issue with the sftp poll-enrich and the post-processing. Please use the loop in one flow and from the loop process call the poll-enrich in another flow connected via ProcessDirect adapter. This solves the issue with the not working post-processing as the flow called via Process Direct is executing the post-processing per file.
Best regards
Mandy
Thank you Mandy Krimmel . Please correct if my understanding is correct?
Main Flow -> Process Call -> ProcessDirect Adapter -> PollEnrich with SFTP
Best Regards
Ramana Reddy
correct
Hi Mandy Krimmel,
Please note, currently there is an issue with the sftp poll-enrich and the post-processing. ---Is this issue fixed in the new patch
Ramana Reddy Reddy Moole
Did the workaround solved your issue.
The discussions about how to solve them are still ongoing. It is unfortunately not so easy. You should still use the workaround as described above.
Best regards
Mandy
Hi Mandy,
Very informative blog!
There is one scenario, I used a looping process call->Poll Enricher->SFTP.(to poll multiple files and to use the functionality of poll enricher).
And there are 2 cases,
Case1.If No file exists in SFTP ,throw an exception when polled files from SFTP at the scheduled date & time.
Sol. To check the checkbox "Throw Exception in no message found" in Poll enricher.
case2. If there are one file/multiple files it should process it and should not through any exception.
In this case I need your help
1.What will be the looping process call condition?
2.And can I use "CamelFileName" header in Looping process call condition?
Also I have raised my question in the below link.
https://answers.sap.com/questions/13913866/how-to-fetch-multiple-files-from-sftp-using-poll-e.html
Please let me your comments.
Thanks,
Purusottam
Hello,
let me give you a suggestion: when polling for the files switch off 'ThrowException', and use the documented header SAP_PollEnrichMessageFound to validate if a file was found and either continue or not. Using the CamelLoopIndex header you can further evaluate if there was no file at all processed and create an exception if required.
Hope this helps.
Best regards
Mandy
Thanks for your suggestion.
BR,
Purusottam
Hi Mandy,
I was wondering when the Poll Enrich step will support the FTP adapter?
Because I am having trouble using FTP adapter currently connected to the start message as it won't let me use the date variable or properties as it is right at the start. I need to access directories depending on the current date.
Thank you for the support in here!
Best regards
Stefan
As there were still some improvements in the Poll-Enrich step required, we had to move the support for FTP adapter to a later time. As soon as we sort out the current issues we will enhance it with the FTP Adapter. But this will be earliest in Q3 2022.
Best regards
Mandy
Thank you for the answer.
So, in general, there is no way yet to dynamically get files for FTP except when using scripts?
Best regards
Stefan
This is correct, script is currently the only option. Once we have the ftp adapter in Poll-enrich this will be possible.
Best regards,
Mandy
Hello Mandy, nice blog! Congrats!
Could you please clarify if there are several files in the directory to be read how is the behaivor of the adapter?
I tried with two files in the same directory and only one of them was read.
Thank you!
Fellipe
This is correct, Poll-Enrich always only polls one file. If you need to poll multiple file you need to do this Poll-Enrich call in a loop.
Best regards
Mandy
Hello Mandy!
Thank you for your attention and quick response.
I tried to implement the Poll-Enrich call in a loop, the first time, a file is read (SAP_PollEnrichMessageFound = true), however, in the second time, the file is not read (SAP_PollEnrichMessageFound = false).
Do I need to clear something (property, header, etc) after the first read?
Thank you!
Fellipe
Hello Felipe,
usually this should work.
I remember that there is/was an issue with the sftp poll-enrich and the post-processing in a loop. Please try to use the loop in one integration flow and from the loop process call the second flow via processDirect adapter. The second flow then contains the poll-enrich with sftp.
Main Flow -> Loop Process Call -> ProcessDirect Adapter -> Second Flow with PollEnrich with SFTP
This solves the issue with the not working post-processing as the flow called via Process Direct is executing the post-processing per file.
Please let me know if this works for you.
Best regards
Mandy
Hello Mandy.
Thank you for your recomendation. It worked in this way.
It's important to say that is not possible to transfer all Headers (set * in the Runtime Configuration - Allowed Header(s)) to the Second Flow. It looks like there is some problem with headers populated from the previous round. At least that's what I realized.
Best regards.
Fellipe
Hello,
some internal or adapter specific headers are not transported, this is correct.
Best regards
Mandy
Hello Mandy.
Thanks for the answer, what I meant is that if you transfer all headers (with *), the second pooling doesn't work, because the headers are transfered with "dirt" from the previous run.
I believe this is also what happens when you run two (or more) pools within the same iFlow (by loop).
It might be interesting to think about creating a parameter in the Pool enrich function allowing you to configure it to clear the headers to perform the second (and nexts) pooling.
Best regards
Fellipe
ok, thank you for this details, will forward this information to the developers.
Best regards
Mandy
Hi Mandy,
I am unable to fetch file using process direct and poll enrich step.
Flow :- Main flow(JMS Sender --> Process Direct)-->2nd flow(Process Direct-->Poll Enrich(pickfile and delete))
Log from Adapter Monitoring which shows the correct filename but file is not picked and replaced with input data using Poll Enrich step.
sap-sftp://virtualSFTP:450/Desktop/SFTP_SRC?authMethod=user&autoCreate=false&connectTimeout=10000&connectivityServiceProvider=%23connectivityServiceProvider&credential=SFTP_USER&delete=true&disconnect=true&fileNameFilter=TestS1.xml&inProgressRepositoryFactory=%23inProgressRepositoryFactory&initialDelay=0&keyStoreAccess=%23keyStoreAccess&knownHostsAccess=%23knownHostsAccess&maxMessagesPerPoll=1&maximumReconnectAttempts=3&noop=false&passwordStorage=%23secureStore&pdUri=&proxyType=onPremise&readLock=none&reconnectDelay=1000&recursive=false&soTimeout=300000&startingDirectoryMustExist=true&stepwise=false&streamDownload=false&synchronous=true&throwExceptionOnConnectFailed=true
Regards,
Sachin Belide
Hello,
I would suggest to open a ticket on LOD-HCI-PI-CON-SOAP so that the experts can check the issue in more detail. Its hard to do troubleshooting without any details. Please attach the message processing logs and the two integration flows to the ticket.
Best regards
Mandy
Hi Mandy,
Thankyou for the quick response.
This scenario did not work in trial tenant. I have imported the same flows in Dev account and it worked.
Regards,
Sachin Belide
that is strange but it's good that you got it working now.
Thank you for the information.
Best regards
Mandy
Hi Mandy,
there are some news about this component?
We asked more feature on it by Influence site and we hope they will take care about them.
1 ) List of Files before download them (check correct number, or use them on a splitter)
body example of output:
<message_enrich>
<file>
<index>1</index>
<filename>File1.xml</filename>
<size>34555</size>
<absolute_path>/in/budget/2022/</absolute_path>
<permissions>755</permissions>
<user_owner>user22</user_owner>
<group_owner>budget_group</group_owner>
</file>
<file>....</file>
</message_enrich>
2 ) Attribute details of a single file or directory (check permissions for validation/error allerting
Federico Bellizia
The features will be discussed further in the implementing team based on priority. It was not yet taken up.
Best regards
Mandy
I did a groovy solution with Jcraft Jsh Library.
Now I'm trying to create an Adapter to use instead SFTP ones with all SFTP commands (ssh commands too).
Federico Bellizia
Hi Mandy,
let's say I get a CSV file from a SFTP server. I noticed that the file doesn't get written into the message body after Poll Enrich received it. I'd like to convert it via CSV to XML in one case and in the other I want to access it via Groovy Script but it doesn't work. How can I make this work? Trying to store it in a data store doesn't work either. It saves the last message to the data store then. I used 'Replace' in the Poll Enrich element.
Thank you
Hi,
don't fully understand your scenario. You use Poll-Enrich to poll a specific file from sftp and then convert it from CSV to XML? Usually this should work. I would suggest you activate trace to check the processing in more detail. Best would be if you could create a ticket on LOD-HCI-PI-CON-SOAP, attach the integration flow and add a clear description of the problem. The experts will have a look and suggest.
Best regards
Mandy
Never mind. I just noticed that the default behavior for the SFTP File processing with Poll enrich is "Delete file". This is a very unlucky default. The file that I wanted to get didn't exist. I couldn't notice because the file name consists of a long string of numbers and there are many files in there. "Keep file and process again" seems to be a more friendly default.
I thought that that must be it. But no, message.getBody() in Scripts now contains the apache camel RemoteFile but if you have a write operation to the data store after the poll enrich it saves the message from before the poll enrich. I create a ticket.
Yes, please create a ticket. Would be great if you can share the solution afterwrds.
Hi
I have a scenario
I have 6files in a folder , let's say 3jpg and 3pdf
I want the output to be 3 zip file. In each Zip file there should be 1pdf and 1jpg.
How can I solve this
Hello,
I could imagine the following solution:
In the integration flow do a parallel multicast fetching a pdf and the corresponding jpg with poll-enrich, in gather use the zip aggregation strategy to zip the files into one zip file:
Cloud Integration – ZIP and Compress Capabilities: Encoder, Decoder, Splitter, Aggregate | SAP Blogs
Please let me know if this works for you
Mandy
Hi,
Can you please help me how to add process direct adapter in loop process call as I want to pull multiple files from sftp with poll enrich.
Hello,
I don't understand your question. Could you explain a bit more about the scenario and were exactly the process direct adapter needs to come into the game?
best regards
Mandy