Skip to Content
Technical Articles
Author's profile photo Kyou Gu

Polling File Data from Web Server with REST Adapter of SAP Process Orchestration

Generally for file based integration, SAP Process Orchestration File Adapter or SFTP Adapter is used to poll the data files from the file system or FTP/SFTP servers. But in some file integration scenario, the web server (Nginx, Apache, IIS etc.) is to be used instead of FTP/SFTP server in case that the FTP/SFTP server is not available. The sender system will put the data files in the web server, and SAP Process Orchestration needs to poll the file data from the web server with HTTP/HTTPS protocol.

This blog post introduces how to use SAP Process Orchestration REST Adapter to poll the file data from the web server. And it explains how REST Adapter will determine what charset is used to poll the file data.

Scenario

We would like to implement an interface with SAP Process Orchestration to poll file data every 24 hours from the Nginx web server and store it in the file system.

The file data format could be JSON, XML, CSV etc.

The charset of the file data could be UTF-8, Shift_JIS, JIS etc.

To make it simple, no data format conversion and no mapping are to be implemented here. Let us focus on the configuration of REST sender channel and how REST adapter will determine the charset used to poll the file data.

Configuration of the REST Sender Channel

In the configuration screen of REST sender communication channel, select REST adapter type, and select REST Polling as the Message Protocol as well. Input the URL of the file to be polled in Target URL, and set the polling interval(seconds).

Select JSON data type in case that the file data is in JSON format.

Select XML data Type in case that the file data is in XML format. And if the file data format is  CSV, fixed length or others, you can select XML data type for it.

How REST Adapter will determine what charset used to poll the file data

When polling non-ASCII file, you may find that the characters of payload data were corrupted in Message Monitor screen. It was caused in case that the wrong charset was used by REST Adapter when polling the non-ASCII file data.

In the configuration screen of REST Polling sender channel, there is no charset parameter provided. So what charset will be used by REST Adapter to poll the file data?

REST Adapter will determine the charset as below when doing REST polling.

  • REST Adapter will always use UTF-8 to poll the data if JSON data type is selected.
  • REST Adapter will use the charset obtained from HTTP Content-Type header of the HTTP response if XML data type is selected.
  • REST Adapter will use the default ISO-8859-1 charset if XML data type is selected and no charset parameter is obtained from HTTP Content-Type header of the HTTP response.

To avoid the character corruption of the non-ASCII file data, the charset parameter needs to be contained in the HTTP Content-Type header of the HTTP response.

Generally the web servers (Nginx, Apache, IIS etc.) provide the parameters to configure the charset in the HTTP Content-Type header of the HTTP response. The below is an example to configure Nginx web server to set Shift_JIS charset in HTTP Content-Type header of the HTTP response when polling file. The charset parameter can be configured in the nginx.conf file of the Nginx web server.

After configuring the above charset parameter and reloading Nginx server, you can use the below curl command to check if the charset parameter has been contained in the Content-Type Header of the HTTP response.

curl -I <URL of file to be polled>

I hope this blog post was helpful to understand how to configure the REST sender channel and how to set charset when polling non-ASCII file data to avoid character corruption.

 

Assigned Tags

      11 Comments
      You must be Logged on to comment or reply to a post.
      Author's profile photo Eurico Borges
      Eurico Borges

      Thanks for posting this information. Specially the details on how to hadle the file charset. It is always tricky do find out the best way to implement it.

      Kyou Gu for transfering binary files, a pdf for example, should we also select in the REST sender channel the XML option in the DataType field ?

      Regards
      Eurico
      Author's profile photo Kyou Gu
      Kyou Gu
      Blog Post Author

      Hi Eurico,

      Currently (PO7.5 SPS23), only JSON and XML data type are supported by REST Adapter with polling. Both of them are text data types, and the binary data type is not supported for REST Polling  yet. So you cannot use REST Adapter to poll binary data file.

      Regards,

      Kyou

      Author's profile photo Denis MUZHZHUKHIN
      Denis MUZHZHUKHIN

      what if server is not yours and you cannot adjust the setting? in my case server returns Content-Encoding: gzip and xml is damaged. what could be done?

      Author's profile photo Kyou Gu
      Kyou Gu
      Blog Post Author

      You can develop a custom Adapter module which read the response content with your desired charset.

      Author's profile photo Denis MUZHZHUKHIN
      Denis MUZHZHUKHIN

      well, it's a bit difficult approach considering the Data Type is set to XML and there is a UTF-8 explicitly set in payload. worth to raise a ticket to SAP? or maybe will try using another adapter

      Author's profile photo Kyou Gu
      Kyou Gu
      Blog Post Author

      REST Adapter will use the charset obtained from HTTP Content-Type header of the HTTP response if XML data type is selected. You have to adjust the web server setting.

      Author's profile photo Denis MUZHZHUKHIN
      Denis MUZHZHUKHIN

      bypassed this strange issue with binaryContent = true parameter as well.

      Author's profile photo Naveen Kumar P K
      Naveen Kumar P K

      Hi Kyou Gu,

      How to get a .zip file using Rest Polling, is it possible to use Rest polling here? could you please help me on this

      Author's profile photo Kyou Gu
      Kyou Gu
      Blog Post Author

      Rest Polling for .zip file or binary file is not supported by now.

      Author's profile photo Denis MUZHZHUKHIN
      Denis MUZHZHUKHIN

      yes, you could set adapter module paratemer binaryContent = true. see SAP note 3043400. actually this amazing parameter solves so many issues...

      the only issue on my patch level is that if you'll turn on logging you will have a java null pointer exception issue. which is probably ok to live with

      Author's profile photo Kyou Gu
      Kyou Gu
      Blog Post Author

      Dear Denis, thanks for the information. The Paramter binaryContent = true can be used to poll binary files. 🙂