Read Lock Strategy and Post-Processing options in SFTP Sender Adapter of SAP Cloud Integration

Hi Folks,

In this blog post, we are going to cover the Read lock strategy and Post-processing options in the processing parameters of the SFTP Sender adapter in SAP Cloud Integration for File to File transmission scenarios.

Problem Statement: During the development of one of the integration flow, I came across a scenario where the source system is placing the file on the SFTP server and made it visible in the directory, and continued to write that file.

Middleware CPI system is scheduled to poll the file from the Source SFTP server every 10 sec.

Hence, as soon as the file is visible on SFTP, CPI polls that file and process it to the target. Due to this, sometimes the empty files also get processed to the target as the writing was not finished and the file is picked.

The resolution for the above problem is to set the Read Lock Strategy option in the Sender SFTP adapter which should prevent reading the file whose writing is still in progress.

By default, the value of the Read lock Strategy is set to None. The other options apart from None are Rename, Done File Expected, and Content Change.

Rename Option: Rename the file on the SFTP server before reading it. We didn’t want to Rename it.

The other two options in the Read Lock Strategy have come as a boon for this issue:

Content Change: In the case of a filename without extension, the best approach would be indeed the “Content Change” option. For the read lock strategy “Content Change”, the SFTP adapter scans the file every now and then to see whether there is a content change in the file, the scan time interval is at least 1 second. If the file content happened to be not changed between 2 scans, the SFTP adapter might treat this file as a final one and then pick it. Please note if we select this option, files cannot be read as quickly as with the other options.

Done File Expected: For the “Done File Expected” approach, the recommendation is that we use the “. done-file” pattern. The .done file signals that the file to be processed is ready for consumption and SAP Cloud Integration flow can pick it. For the read lock strategy “Done File Expected”, the SFTP team needs to perform the changes in the files and generate the .done file as well in pair with an incoming file when the writing is finished.

There are 2 scenarios for the filenames:

1. Filenames WITH extension

For the filenames with extension, In the Done File Name option, we should give value as ${file:name}.done

2. Filenames WITHOUT extension

For the filenames without extension, In the Done File Name option, we should give the value as ${file:name.noext}.done

Recommendation from SAP standard documentation – We should use the default naming pattern to avoid any uninvited issues.

The next important configuration in the processing tab for SFTP Adapter is Post Processing Parameter.

For the Post-Processing parameter, we have the following options to select as value:

The default value is Delete File

1. Delete File: If we have selected Done File Expected as Read Lock Strategy in pair with the Post Processing value as Delete File, the incoming file to be processed as well as the done file will be deleted.
2. Keep File and Mark as Processed in Idempotent Repository: If we have selected Done File Expected as Read Lock Strategy in pair with the Post Processing value as Keep File and Mark as Processed in Idempotent Repository, an entry will be created in the idempotent repository; the done file will not be deleted.
3. Move File: If we have selected Done File Expected as Read Lock Strategy in pair with the Post Processing value as Move File, only the incoming file to be processed is moved and the done file will be deleted.
4. Keep File and Process Again: For this option, the file processing will be repeated even in case it has not been changed.

Hope this blog post would help you while configuring of SFTP adapter in Cloud Integration.

Happy Development!