Technical Articles
Dynamic Setting of Archive Directory for Post Processing in SFTP Sender Adapter
In this blog, how to move file to different locations after processing is finished is demonstrated. This blog also serves as answer to question: Move original file to another folder on SFTP server in case of exception.
The structure of this blog is as follows. First, requirements from the question are specified. Then, dummy scenario is created to fulfil requirements. Finally, solution is demonstrated.
Requirements
- Files in ‘input’ directory must be processed
- When processing is successful, the processed file must be moved to ‘output’ directory
- When processing is failed, the processed file must be moved to ‘error’ directory
Scenario
XML files will be read from ‘input’ directory. XML files will contain single node ‘Payload’ having single sub-node called ‘Success’. If ‘Success’ is true, processing will be successful. If ‘Success’ is false, process will end with Error End.
Possible Payloads
Success
<Payload>
<Success>true</Success>
</Payload>
Failure
<Payload>
<Success>false</Success>
</Payload>
Solution
The solution uses Content Modifiers to set the property ‘archiveDirectory’. In successful scenario, ‘archiveDirectory’ is set to ‘/output’. In failure scenario, Exception Subprocess is used to catch the exception and Content Modifier is used therein to set ‘archiveDirectory’ to ‘/error’.
Flow Steps
Source for SFTP
Here, Source > Directory is set to ‘/input’ as per requirements. Please fill in Address and Credential Name as per your configurations.
Success? Router
The Success? Router routes based on Success property’s value. If ‘/Payload/Success’ is ‘true’ then ‘archiveDirectory’ property is set to ‘output’ otherwise flow ends with Error End.
Set Archive Directory to ‘output’
If successful, ‘archiveDirectory’ property is set to ‘output’.
Set Archive Directory to ‘error’
If failed, ‘archiveDirectory’ property is set to ‘error’.
Processing for SFTP
After processing is complete (whether succesfully or not), the file is moved to a location specified dynamically using ‘archiveDirectory’ property.
Parameter | Value |
Post-Processing | Move File |
Archive Directory | /${property.archiveDirectory}/${file:name} |
Execution
For testing this flow, Success.xml and Failure.xml files, containing payloads specified in Possible Payloads section above are put in ‘input’ directory.
After flow polls files, Success.xml file is put in ‘output’ directory and Failure.xml file is put in ‘error’ directory.
Alternate Scenario(s)
Error occurs when sending the file to receiver SFTP Server
In this scenario, file is sent to SFTP server. However, ‘Create Directories’ option on SFTP Receiver is disabled and the directory in which file must be put is not created on server to deliberately cause exception.
Integration Process
SFTP Receiver Target Configuration
Please fill in Address and Credential Name as per your configuration.
SFTP Receiver Processing Configuration
Note that ‘Create Directories’ option is disabled.
Example Run
1. Input
File ‘Non Existing Directory.xml’ is uploaded to ‘input’ directory. The file contents are same as Success under Possible Payloads above.
2. Success? Router
As ‘/Payload/Success’ is ‘true’, ‘Yes’ branch is followed.
3. Set Archive Directory to ‘output’
Exchange Property ‘archiveDirectory’ is set to ‘output’.
4. Send file to SFTPReceiver
File is sent to SFTPReceiver.
5. Error occurs because the directory ‘nonexisting’ does not exist
As ‘nonexisting’ directory does not exist, exception is thrown by SFTP and is caught in CPI as:
org.apache.camel.component.file.GenericFileOperationFailedException: Cannot change directory to: nonexisting, cause: 2: No such file or directory
6. Exception Subprocess is triggered
As exception occured, Exception Subprocess is triggered.
7. Set Archive Directory to ‘error’
Exchange Property ‘archiveDirectory’ is set to ‘/error’.
8. Exception Subprocess ends with Message End and triggers the Processing tab of SFTP Sender
In the processing tab, the configuration for Post Processing is to Move File to /${property.archiveDirectory}/${file:name}. As per these configurations, SFTP Sender Adapter moves the read file from ‘/input’ directory to ‘/error’ directory.
9. Output
Conclusion
As demonstrated, file can be moved to different locations when processing is finished using dynamic path. This is achieved when integration process ends with Message End i.e, message is completed either through success path or error path via Exception Subprocess, The exchange property ‘archiveDirectory’ was set at runtime and used to specify path where file should be moved dynamically.
Hi Bala,
Hope you are doing good.
I have a requirement to read the directory details dynamically and pass into mapping as a input for one of the filed in CPI.
could you please guide me the camel expression to fetch the folder details dynamically from the source SFTP.
Thanks in advance!
-Avinash
Hi RVS Avinash Varma Nadimpalli,
Check the File Language in Apache Camel documentation. If it doesn't help, you can ask a question to SAP Community.
Hope this helps,
Bala
Hi Bala ,
Thank you for the wonderful explanation . Would just like to check , in normal real time XML payload , what should I check for a condition to see if my payload is successful or in error ? To move the files accordingly to the folders .
Appreciate your help .
Thanks ,
Nitheesh
If you receive payload in CPI, then you can route to Error End if the value of certain element indicates the error using XPath.
Hope this helps,
Bala
Hi Bala,
Hope you are doing good.
Could you please provide your inputs on below issue.
https://answers.sap.com/questions/13090745/move-original-files-to-another-folder-on-sftp-serv.html
Tried with above approach but unable to do as we are having multiple files to move and only one final RFC connection.
Thanks,
Avinash
Hello Bhalchandra,
Nice blog!
If I am not wrong, in this case, iFlow status in CPI monitoring will always be successful.
In case of error, is there any way to fail iFlow status and still move file to error folder?
Regards,
Vinod Patil
Thanks, Vinod Patil.
As per Help Document, only successful messages trigger Post Processing. So, Post Processing is not an option for your requirement.
You can use the combination of Bram's comment and Sriprasad's answer like so:
Hope this helps,
Bala
Hi Vinod Patil,
Hope you are doing good. are you able to achieve this. ,
Hi Bhalchandra Wadekar,
Thanks for the wonderful blog and help.
I have the same requirement and i stuck in deleting the file.Able to move the files to error folder but unable to delete.
what would be the input adapter for second flow in the above provided solution ?
Thanks,
Avinash
The second flow will start on schedule using the Timer Start Event.
Hope this helps,
Bala
Hi Bala,
Thanks for the help.
It worked with process direct.
Thanks,
Avinash
Hi Avilash,
I did try with Process Redirect but it still fails my first flow in case my second flow (subscription flow) errors out which again will not delete my source file.
Could you let me know if there is any specific option you used to delink the processing of second flow with first flow. Basically i want the first flow to be successful once the process redirect successfully makes call to next flow and if there is any error in second flow i have the exception sub process defined there it self.
Thanks and Regards
Appasab
Hi Bala, works great, thanks, but what if there is an exception in the exception subprocess, e.g. in error subprocess ‘archiveDirectory’ property is set to ‘error’ and email is sent with the exception information. Sending email fails for some reason (noticed with quota issue on the test O365 account). Because successful messages trigger Post Processing, postprocessing fails. This need something like nested exception subprocess or storing contents in Data Store?
Hi Sven Kelemen,
Nested exception handling is not supported.
You could achieve the scenario you mentioned in another way by having a Local Process Call in the main Exception Subprocess. In the Local Integration Process, you can send an email. If sending the email fails, the Exception Subprocess inside the Local Integration Process will handle the error and end with End Message Event. Please ensure that, in the main Exception Subprocess, you have set the directory to desired error directory before calling the Local Integration Process.
Exception Handling with Email Notification
You could achieve the same using a separate Integration flow to send an email and calling it using ProcessDirect Adapter. Remember to handle exceptions in this process and end with an End Message Event or an Escalation End to continue processing in the main subprocess.
Hope this helps,
Bala
P.S.: I don't have access to SFTP now. So, can't test this solution. Please confirm if the solution helps and works for you for the benefit of future readers :).
Hi Bala, it works as required, file is moved to property.archiveDirectory e.g. 'error' even when sending mail fails. Thanks. But it would be good to get error in the Message Monitor. Now status is 'Completed' meaning error information is lost. Kind regards, Sven
Hi Sven Kelemen,,
Thank you for confirming the solution works. I am glad :).
Could you please try using Escalation End in the Exception Subprocess of the Local Integration Process?
Exception Handling with Escalation End
Your requirement and the behaviour of Escalation End is explained with an example in the blog Ending the message the right way!
Kind regards,
Bala
Hi Bala, thanks again, this is better, now getting "Escalated" instead of "Failed" or "Canceled" which would be preferred.
Hello,
Just wanted to add that I was having a lot of trouble getting this to work. If I hardcoded 'error' into the field Archive Directory, it worked fine, but as soon as I set the property archiveDirectory to 'error', it wasn't working (the file just stayed where it was).
In the end, this did work: ${property.archiveDirectory}/}/${file:onlyname.noext}.${file:ext}
The file coming from the SFTP was a CSV, not an XML (no idea if this is the reason the solution described in the blog post wasn't working for me).