Polling File Data from Web Server with REST Adapter of SAP Process Orchestration
Generally for file based integration, SAP Process Orchestration File Adapter or SFTP Adapter is used to poll the data files from the file system or FTP/SFTP servers. But in some file integration scenario, the web server (Nginx, Apache, IIS etc.) is to be used instead of FTP/SFTP server in case that the FTP/SFTP server is not available. The sender system will put the data files in the web server, and SAP Process Orchestration needs to poll the file data from the web server with HTTP/HTTPS protocol.
This blog post introduces how to use SAP Process Orchestration REST Adapter to poll the file data from the web server. And it explains how REST Adapter will determine what charset is used to poll the file data.
We would like to implement an interface with SAP Process Orchestration to poll file data every 24 hours from the Nginx web server and store it in the file system.
The file data format could be JSON, XML, CSV etc.
The charset of the file data could be UTF-8, Shift_JIS, JIS etc.
Configuration of the REST Sender Channel
In the configuration screen of REST sender communication channel, select REST adapter type, and select REST Polling as the Message Protocol as well. Input the URL of the file to be polled in Target URL, and set the polling interval(seconds).
Select JSON data type in case that the file data is in JSON format.
Select XML data Type in case that the file data is in XML format. And if the file data format is CSV, fixed length or others, you can select XML data type for it.
How REST Adapter will determine what charset used to poll the file data
When polling non-ASCII file, you may find that the characters of payload data were corrupted in Message Monitor screen. It was caused in case that the wrong charset was used by REST Adapter when polling the non-ASCII file data.
In the configuration screen of REST Polling sender channel, there is no charset parameter provided. So what charset will be used by REST Adapter to poll the file data?
REST Adapter will determine the charset as below when doing REST polling.
- REST Adapter will always use UTF-8 to poll the data if JSON data type is selected.
- REST Adapter will use the charset obtained from HTTP Content-Type header of the HTTP response if XML data type is selected.
- REST Adapter will use the default ISO-8859-1 charset if XML data type is selected and no charset parameter is obtained from HTTP Content-Type header of the HTTP response.
To avoid the character corruption of the non-ASCII file data, the charset parameter needs to be contained in the HTTP Content-Type header of the HTTP response.
Generally the web servers (Nginx, Apache, IIS etc.) provide the parameters to configure the charset in the HTTP Content-Type header of the HTTP response. The below is an example to configure Nginx web server to set Shift_JIS charset in HTTP Content-Type header of the HTTP response when polling file. The charset parameter can be configured in the nginx.conf file of the Nginx web server.
After configuring the above charset parameter and reloading Nginx server, you can use the below curl command to check if the charset parameter has been contained in the Content-Type Header of the HTTP response.
curl -I <URL of file to be polled>
I hope this blog post was helpful to understand how to configure the REST sender channel and how to set charset when polling non-ASCII file data to avoid character corruption.
Thanks for posting this information. Specially the details on how to hadle the file charset. It is always tricky do find out the best way to implement it.
Kyou Gu for transfering binary files, a pdf for example, should we also select in the REST sender channel the XML option in the DataType field ?
Currently (PO7.5 SPS23), only JSON and XML data type are supported by REST Adapter with polling. Both of them are text data types, and the binary data type is not supported for REST Polling yet. So you cannot use REST Adapter to poll binary data file.
Hi Kyou Gu,
How to get a .zip file using Rest Polling, is it possible to use Rest polling here? could you please help me on this
Rest Polling for .zip file or binary file is not supported by now.
When using incremental request - it only fetches one messge per polling.
This renders the incremental feature useless.
The just fetched message is not confirmed until hte next poll which is too late. You expect the fetched message to be confirmed immediately.
Is there some way to make it fetch all messages at once until there are no more messages to fetch?