Skip to Content
Technical Articles
Author's profile photo Alexander Bundschuh

PI REST Adapter – Polling a REST API

This blog is part of a collection of blog entries that shows architectural concepts and configuration of the SAP PI REST Adapter. We also added some sample scenarios to make it easier for you to understand how your scenario can be implemented using the PI REST Adapter.

If you haven’t done so far, best is to start with the very first blog PI Rest Adapter – Don’t be afraid within the blog series covering the concepts of the REST adapter. A complete list of all blog entries can be accessed from here PI REST Adapter – Blog Overview.

The current blog describes how to poll a REST API. The sender REST adapter in polling mode has been supported since release 7.31 SP16 / 7.4 SP11.

Scenario

We would like to frequently poll video information from Google’s Youtube REST API, and store the same on a file system. Furthermore, the resultset should be split into individual messages per video ID, and already polled results should be discarded.

In the SAP Process Integration Designer perspective of the NetWeaver Developer Studio (NWDS), I have defined an Integration Flow with a REST sender channel and a file receiver adapter.

01 iflow.png

Let’s focus on the configuration of the sender channel.

Configuring the REST sender channel

Double-click on the sender adapter to open the channel editor. Select the REST adapter type, and from the Message Protocol drop down menu the entry REST Polling.

02 Adapter type.png

Option 1: Incremental request based on timestamp of last call

In order to better understand the configuration below, let’s take a look at a sample response in JSON format. The REST API returns an array of items each having a unique ID stored in the etag field.

05 sample json response.png

First switch to sub-tab Data Format below tab Adapter-Specific.

The format is JSON, so select JSON from the Data type drop down menu. We would like to convert the JSON into XML, so select the Convert JSON to XML check box, and add a wrapper element to ensure that the converted XML format contains one root element only.

As mentioned below, the message should be split into individual messages per video ID. Select Split Result into Multiple Messages check box. As Array Containing Messages maintain items (See message format above).

Furthermore, duplicates should be removed. Select the Filter out Duplicates check box, and maintain etag as Unique ID Element (See message format above).

To place incremental requests, we would like to use the timestamp of the latest call. The value is stored between the calls and can be used in the REST URL as a placeholder with name incrementalToken. By the way, alternatively you can also use an XPath expression or a JSON element from the response of the last call such as next Page indicator or similar (shown below). From the Incremental Type drop down select entry Timestamp of Last Call. The timestamp format of the API complies with ISO 8601, and needs to be specified as follows: yyyy-MM-ddTHH:mm:ssZ with T and Z being constants. The REST adapter follows the Joda time format, see DateTimeFormat (Joda time 2.2 API). So, we need to place the constants in quotes. Maintain the Timestamp Format attribute as yyyy-MM-dd’T’HH:mm:ss’Z’, and define an initial value that complies with the format.

04 data format.png

Switch to tab HTTP Request, and maintain the Target URL as follows: https://www.googleapis.com/youtube/v3/search?key=<your API key>&part=id&q=SAP&maxResults=50&order=date&publishedAfter={incrementalToken}. Note, I added the placeholder incrementalToken in curly brackets which holds the timestamp of the last call. As HTTP Operation select GET. Finally, define a polling interval, here 3600 seconds.

03 http request.png

Option 2: Incremental request based on response content

As another option I would like to show you how to configure the incremental request in case that many pages need to be requested. As you can see below, in the response two additional elements are added holding the previous (prevPageToken) and the next page token (nextPageToken). Latter can be used in the URL to gather the next response page.

01 Chrome.png

From the Incremental Type drop down select the entry Response Content, and maintain the incremental token element as nextPageToken. This will store the value of the nextPageToken element into the placeholder incrementalToken.

From release 7.31 SP17 / 7.4 SP12 / 7.5 SP01 onwards, you can define an action in case that the very last page has been reached and hence the nextPageToken element is missing. You have the following options:

  • treat as error which is the default
  • skip current poll and retry later
  • use an empty token value
  • use the initial token value
  • use a custom token value

Here, we chose the option Use Empty Value for Token.

02 Config.png

On tab HTTP Request, maintain the Target URL as follows: https://www.googleapis.com/youtube/v3/search?key=<your API key>&part=id&q=SAP&maxResults=5&pageToken={incrementalToken}. Note, I added the placeholder incrementalToken in curly brackets which holds the token of the next page.

03 Config.png

I hope this blog was helpful to understand how to configure the REST sender channel in polling mode. If you like to learn more, check out the other blogs in the series, accessible from the main blog PI REST Adapter – Blog Overview.

Assigned tags

      39 Comments
      You must be Logged on to comment or reply to a post.
      Author's profile photo Former Member
      Former Member

      Hi Alex,

      Thanks for the Blog, it is very good to Understand

      However I have still confusing on my Target URL

      This is my target URL

      https://tools-dev:9307/ceres/data/basic?lastChangedTime=2015-06-08T23:00:00Z

      The value must be an ISO 8601 formatted time, containing the time zone, e.g. 2015-06-08T11:30:00Z



      So how my URL look like with place holder? please suggest me.


      Thanks,Sateesh

      Author's profile photo Alexander Bundschuh
      Alexander Bundschuh
      Blog Post Author

      Hi Sateesh,

      similar to my sample in the blog

      url: https://tools-dev:9307/ceres/data/basic?lastChangedTime={incrementalToken}

      timestamp format: yyyy-MM-dd'T'HH:mm:ss'Z'

      Alex

      Author's profile photo Former Member
      Former Member

      Hi Alexander,

      I did same way on earlier also.

      and I get this error

      Error while processing inbound message. java.lang.RuntimeException: java.net.URISyntaxException: Illegal character in query at index 83: https://tools-dev:9307/ceres/data/basic?lastChangedTime={incrementalToken}: java.net.URISyntaxException: Illegal character in query at index 83: https://tools-dev:9307/ceres/data/basic?lastChangedTime={incrementalToken}: Illegal character in query at index 83: https://tools-dev:9307/ceres/data/basic?lastChangedTime={incrementalToken}


      Thanks & Regards,

      Sateesh

      Author's profile photo Former Member
      Former Member

      Hi Alex,

      Our SP16 REST adapter features is supporting to convert from JSON to  XML with Multiple data sets?

      I heard from some threads those SP14 & SP15 are not supporting multiple datasets when converting from JSON to XML

      if this is also same case on SP16 , then how can I  proceed to accepts multiple Datasets.

      Please suggest me on this.

      Thanks & Regards,
      Sateesh N

      Author's profile photo Alexander Bundschuh
      Alexander Bundschuh
      Blog Post Author

      Hi Sateesh,

      the REST adapter does support JSON arrays for both JSON to XML and XML to JSON conversion. There is only one restriction when an XML element which is unbounded contains only one occurence, it won't add the brackets [ and ] to the JSON, if there are at least two entries it will. Some REST APIs need the arrays even if the element occurs only once. This will be addressed with SP17.

      Alex

      Author's profile photo Former Member
      Former Member

      Thanks Alex for clarification:)

      Author's profile photo Former Member
      Former Member

      Hello Alex,

      I am using this URL to call REST webserive using Rest Sender poll adatper

      url: https://tools-dev:9307/ceres/data/basic?lastChangedTime={incrementalToken}


      In the Incremental Request tab


      IncrementalRequest.jpg

      If I used Time stamp format  is this way yyyy-MM-dd'T'HH:mm:ss'%2B02:00' then only PI retrieved latest changes from REST service.

      If I used Time stamp format is this way yyyy-MM-dd'T'HH:mm:ss'Z', PI doesn't receive changes from REST webservice

      Currently CET time is 2 hours ahead into UTC time, after october this year again CET will be changed and at that only 1 hour ahead into UTC time. If that is the case I need to change a communication channel twice a year.

      My complain is I do not want change the communication channel twice a year, can you please give me some idea, how I can put permanent Time Stamp format in the Incremental Requests?

      Please suggest me.

      Thank you

      Sateesh

      Author's profile photo Former Member
      Former Member

      Hi Alex,

      Polling is a great feature within REST Sender adapter. However there appears some issue with HTTP header parameters. Though we have set the parameter. REST sender adapter is not passing that to the remote system. We are trying to pass some system specific information and the call is failing because the header parameters is not passed.

      Is this is a known issue? Are there any workaround or any patch available. We are currently on the latest SP 11.

      I appreciate any response from you.

      Thanks

      -Pradeep

      Author's profile photo Ashutosh R
      Ashutosh R

      Hi Pradeep,

      did your problem got solved, as we are also facing issue with rest sender polling  with adding custom header to be passed with request message.

      regards,

      ashutosh

      Author's profile photo Alexander Bundschuh
      Alexander Bundschuh
      Blog Post Author

      Hi Ashutosh, Pradeep,

      this was a bug and should be solved with the latest patch, if you are on the latest patch and still face the issue, please raise an incident ticket

      Alex

      Author's profile photo Peter Chezowitch
      Peter Chezowitch

      Hi Alex,

      Passing own Header variables with the polling request seems to work now. According to the documentation, response header variables from the REST Sender with polling should also be available as XI-Headers (similar to the receiver channel). I have tested this on two 7.5 systems with current SPs and this does not seem to be the case. There are no XI headers present inside the rest namespace from the sender channel with polling.

      Is there a fix for this already? I have not found one when browsing the notes.

      Author's profile photo Former Member
      Former Member

      Thanks Alex,

      It is was a bug and i had opened a incident ticket and it is resolved in following note.

      29.10.2015 - 10:34:30 EST - Reply by SAP

      Hello,
      The issue is corrected for 7.40 SP11 with SAP Note 2202836 #REST
      Poller channel ignores custom headers#

      '

      We did apply this note and it has indeed fixed the issue.

      -Pradeep

      Author's profile photo Former Member
      Former Member

      I had below queries on the REST Polling adapter :

      1. How do you view the runtime target HTTP URL, do you see in audit logs, I didn't find that. So if I use {IncrementalToken} with Last datetimestamp, I would like to see what value gets replaced in the token each time the interface runs. If the call errors out I can see the error URL but not when it is a success I can't see the URL

      2. In " Timestamp of last call " there is a field for Initial value, when does this get into effect. The first time we run this interface ? Say I am stopping this channel and again restarting it, then what time it takes for " Timestamp of last call " or does it take the initial value which I mention in configuration?

      3. There is an option of MessageSpilt - Split Result into Multiple Messages, here I can specify the xpath of the element or table which iterates. So how does this spilt happen, does it slit say I have 5000 records into batches of 50 or it is 5000 single records ?

      Author's profile photo Former Member
      Former Member

      Hi Alex,

      Is there any custom way of handling below HTTP polling scenario. I am trying to get candidates from a Recruiting system and I need to pass below values in URL:

      https://api.jobvite.com/api/v2/candidate?api=<key>&sc=<secret>&format=json &start=1&count=50&datestart=2014-01-01&dateend=2014-07-01

      1. So my variables are "start" which would the next starting index i.e. 1...51...101

      2. datestart would be the last time I ran this interface

      3. dateend would be current time.

      I tried the incrementalcount approach but how do I handle current date also how do I handle both incremental date and nextPageToken ?

      Thanks in advance

      Ravijeet

      Author's profile photo Former Member
      Former Member

      Hi Alex,

      I was trying to check how do we implement the below scenario using incremental token:

      If we are iterating the records page by page, say we have 2000 records and max we can get in one request is 500, the 4 request call will be as below:

      https://api.jobvite.com/api/v2/candidate?api=api_key&sc=abc&start=1&count=500

      https://api.jobvite.com/api/v2/candidate?api=api_key&sc=abc&start=501&count=500

      https://api.jobvite.com/api/v2/candidate?api=api_key&sc=abc&start=1001&count=500

      https://api.jobvite.com/api/v2/candidate?api=api_key&sc=abc&start=1501&count=500

      What value do we store in nextPageToken ?

      Thx in advance

      Ravijeet

      Author's profile photo Former Member
      Former Member

      Hi Alex,

      Do you know whom from the SAP Product development team to reach to discuss above feasibility.

      Regards

      Ravijeet

      .

      Author's profile photo Bhargava krishna Talasila
      Bhargava krishna Talasila

      Hi Ravijeet,

       

      I think there is no standard way to achieve your requirement,you may need to write adapter module.

       

      I suggest you contact Udo Paltzer . He may help you to address your requirement.

       

      Regards

      Bhargava Krishna

      Author's profile photo Former Member
      Former Member

      Hello,

      we have PO 7.4 SP12 and the Rest Polling Protocol is not able to choose in the Communication Channel.

      Do you have any Idea?

      Thanks and regards,

      Fabian

      Author's profile photo Former Member
      Former Member

      Hi,

      We have the same issue as Fabian with same PI version 7.40 and SP12. The 'REST Polling' Message Protocol is missing in the sender adapter.

      Does anybody know if this feature is still supported with 7.40 SP12?

      Thanks,

      Filipe

      Author's profile photo Former Member
      Former Member

      Hi all,

      Experimenting with incremental request based on response content, I couldn't figure how to deal with XPath expression and ATOM feeds.

      I asked there :REST polling, Atom feed and XPath to incremental token

      I hope someone will enlighten me.

      Thanks in advance,

      Manu.

      Author's profile photo Frederick-Claud Dimmer
      Frederick-Claud Dimmer

      Hi,

      I want to poll yesterdays performance data from our PO using the performancedataqueryservlet. I’m passing in the central adapter engine as component. In addition to that I need to specify a begin and end parameter. The request URL looks as follows:

      http://<host>:<j2eeport>/mdt/performancedataqueryservlet?component=<component>&begin=2016-10-17%2002:00:00.0&end=2016-10-18%2002:00:00.0

      I wanted to use the incremental timestamp but only one timestamp is supported. Is there any solution to this problem?

      Author's profile photo Rafael Vieira
      Rafael Vieira

      Hi Frederick,

      Did you get an answer for this question?
      I'm currently setting a REST sender which has to poll an API and I need to use Begin Date and End Date in its URL.

      Can you share your findings, please?

      Tks.

      Author's profile photo Frederick-Claud Dimmer
      Frederick-Claud Dimmer

      Hey Rafael,

      I could not find a satisfying answer to my question. SRY.

      Author's profile photo Rafael Vieira
      Rafael Vieira

      Hi,

      I need to call an URL as follow:
      http://<host>/rest/<service>/getVehiclesbyDateRange?key=[hard_coded_key]&beginDate=2016-12-09&endDate=2016-12-10

      Begin and End date must be the same so the service would return the list of vehicles for that particular day, only.
      But Begin/End Date must be automatically incremented in each day, always to get the current day's vehicle list.

      Can anyone suggest how to achieve this in the REST Sender with poll?

      Tks!

      Author's profile photo Sara Schmidt
      Sara Schmidt

      Hi Rafael,

      have you resolved your problem?

      Anyway, the URL of my scenario does not work with yyyy-MM-dd'T'HH:mm:ss'Z', but just yyyy-MM-dd.

      Do you know how to get it?

      Thanks and regards

      Sara

       

      Author's profile photo Former Member
      Former Member

      I am getting an error while Polling rest APi like unsupported Media type. in below service

      https://my307381.crm.ondemand.com/sap/bc/srt/scs/sap/yy635qjbly_zendeskticketsin.

      Author's profile photo Dheeraj Kumar
      Dheeraj Kumar

      Hello,

      Please help to resolve below issue.

      Scenario = REST Webservice --> PO --> ECC

      I have configured Sender REST Polling Adapter. Every 30 minutes this adapter starts polling sales order from REST Service. I have selected parameter "Filter out duplicates". So every time it polls it picks up only unique sales order. From morning it works fine but suddenly at 4:30 PM poll time it picks up all the sales order irrespective of duplicates.

      Please help to resolve this issue as this is a production issue.

      Regards,

      Dheeraj kumar

      Author's profile photo Eric Hernandez
      Eric Hernandez

      Hi Alex,

      How can I manage the Incremental Content, when you have a Query in the URL?.

      Example:

      Original URL:

      https://XXXYYYZZZ.salesforce.com/services/data/v43.0/query/?q=select Id, Name,  from Product2&nextRecordsUrl={incrementalToken}

       

      Response 1:

      “nextRecordsUrl”: “/services/data/v43.0/query/01g2900000c6HnhAAE-2000”

       

      Expected Response 2:

      “nextRecordsUrl”: “/services/data/v43.0/query/01g2900000c6HnhAAE-4000”

       

      And so on.

       

      The next execution should be like:

      https://XXXXXXXXX.salesforce.com/services/data/v43.0/query/01g2900000c6HnhAAE-2000

      At the moment I have been unable to make it work. I’m doing tests with:

      Incremental Type: Response Content

      Incremental ID Element: nextRecordsUrl

      Action for Missing or Empty Token: Use Initial Value for Token

      Value: 0

       

      With this, the channel always retrieves information for the first batch as the ID of the "nextRecordsUrl" changes on each execution instead of being maintain.

       

      Example:

      1st time is  "01g2900000c6HnhAAE-2000”

      2nd time is "01gK0000011AgxaIAC-2000” (the ID Changed)

      and so on...

       

      Please advice ?

       

      Thanks and Best Regards,

      Eric

      Author's profile photo Eric Hernandez
      Eric Hernandez

      Hi Alex,

       

      I got it working for two consecutives executions, nevertheless, its always failing on the third one….

       

      The current settings are:

      Tab HTTP Request

      Target URL: https://XXXXYYYY.salesforce.com{incrementalToken}

       

      Tab Data Format

      Incremental Type: Response Content

      Incremental ID element: nextRecordsUrl

      Action for Missing or Empty Token: Use Initial Value for Token

      Initial Value: /services/data/v43.0/query/?q=select Id, Name, ProductCode, from Product2

       

      With these configuration the results were the followings:

      1st Execution

      <totalSize>4952</totalSize><done>false</done><nextRecordsUrl>/services/data/v43.0/query/01gK0000011AlBCIA0-2000</nextRecordsUrl>

      2nd Execution

      “><totalSize>4952</totalSize><done>false</done><nextRecordsUrl>/services/data/v43.0/query/01gK0000011AlBCIA0-4000</nextRecordsUrl>

      So far it is working propertly, but when getting to the last execution the JSON do not have the field "nextRecordsUrl"

      Last Execution

      “Fatal error while proccessing inbound message.com.sap.aii.af.lib.mp.module.ModuleException: com.sap.aii.adapter.rest.ejb.parse.InvalidJSonPath: JSON path “nextRecordsUrl” could not be found.

      Any suggestion regarding why is always failing during the third execution will be highly appreciated.

       

      Thanks and Best Regards,

      Eric

      Author's profile photo Alexander Bundschuh
      Alexander Bundschuh
      Blog Post Author

      Hi Eric,

      it's actually possible to define an action if the result does not contain the expected information, here nextRecordsUrl, see https://help.sap.com/viewer/5cf7d2de571a45cc81f91261668b7361/7.5.13/en-US/d4ee3eca7baf436b996e2473c04809b4.html

      Alex

      Author's profile photo Adam Filipiak
      Adam Filipiak

      Hi Alex,

      question about feasibility.

      I have such scenario:

      Web service to be pooled once a day with dynamic URL, excactly this https://www4.bcb.gov.br/Download/fechamento/20190409.CSV
      The date of file in the URL is dynamic and needs to be {today – 1}.
      The response file content is in CSV and needs to be passed as an IDoc to ERP

      Can this be done with REST or SOAP/Axis will be better for this?

      If this is not possible then I will mabye invoke the webservice from ERP with ABAP Proxy and use Async/Sync bridge in REST adapter.

      Thanks in advance ?
      Adam

      Author's profile photo Sara Schmidt
      Sara Schmidt

      Hi Adam,

      I have the same requirement as yours.

      Have you got solution how to set date (current date -1) in the URL?

      Thanks and regards

      Sara

      Author's profile photo Adam Filipiak
      Adam Filipiak

      Hi Sara,

       

      partialy only ;).Now I'm using REST polling sender with incremental pattern set to [today - 1] date.

      It works generally but fails on holidays when the exchange rate is not puiblished and file is not available for the adapter. This is also the case for Saturdays and Sundays but this I handle in adapter activity scheduling. So, shortly - I'm still working on that.

      My next idea is to use AXIS with some module / handler which will populate the proper date in the URL.

      But this means java coding. Sooner or later I have to solve that. Stay tunned 😉

      Author's profile photo Bartlomiej Krawiec
      Bartlomiej Krawiec

      Adam Filipiak Would you please share how you achieved to make 'today - 1' scenario work?

      Author's profile photo Adam Filipiak
      Adam Filipiak

      Hi Bartłomiej,

      sorry for late response. I should check my inbox more often ;).
      Below is my temporary & lame solution to the “today – 1” problem.

      Upon chanel activation you have to provide the initial value = “today – 1” and thats the error prone, lame part ;).

       

      Author's profile photo Vijay Kamath
      Vijay Kamath

      Hi Adam,

       

      I need to pass 2 parameters in the REST URL - start_time and end_time (in the format 2020-06-16T08:00:00.000-05:00) and increment by 1 hour. How can I pass these 2 parameters in the REST Polling Sender channel?

      Author's profile photo Sara Schmidt
      Sara Schmidt

      Hi Alexander,

      great blog!

      Actually my scenario works well with Rest-Polling with get operation.

      Just two problems disturbing me.

      The first is the Encoding of the response, which contains a csv file in the XML payload and has codepage ISO-8859-1. The german characters in the csv file become devious. Do  you know how to set character-set of response?

      The second is the incremental timestamp. Actually my senario just needs yyyy-MM-dd. But it does not work.

      I am appreciated if you could give me any hints.

      Regards

      Sara

       

      Author's profile photo Vijay Kamath
      Vijay Kamath

      Hi Sara,

      I am facing a similar issue where I need to pass 2 parameters in the URL for REST Polling - start_time and end_time in the following format - YYYY-MM-DDTHH:MM:SS.000-05:00 (Ex. 2020-06-16T08:00:00.000-05:00).  Any pointers on how to pass 2 parameters in the channel would be very helpful.

      Thanks,

      Vijay

       

      Author's profile photo Otto Frost
      Otto Frost

      regarding

      Option 2: Incremental request based on response content

       

      • Only one message is fetched per polling interval. You would expect that if the poll fetched a message then it should immediately poll again and fetch next message. It shouldn' wait poll interval. When poll fails to fetch a message it should wait poll interval.

       

      • In request the incremental token can only be sent as url parameter. You would like to have the possibilty to send as http header or json/xml/text in the http body.

       

      • In the response it would be good to have binary support and then the incremental token in http header. For performance reason we don't want incremental token in the body because it is expensive to  parse the body.

       

      • The regular rest without polling supports multipart attachments which is needed for polling too.

       

       

      • Maybe you want to use this service below - but the po adapter can't do it. The po adapter will only poll once fetch one event and the wait poll interval instead of fetching all events immediately. Hope sap has plans to improve the adapter.
        https://docs.oracle.com/en/cloud/saas/field-service/18c/cxfsc/op-rest-ofsccore-v1-events-get.html

       

       

      • When calling an azure function or logic app you need to set <SECRETKEY> in the url, example below.
        In PO the <SECRETKEY> will be visible in the gui as there is no way to mask it with **** on polling adapter. In module chain you can hide secrets using the pwd.<password> syntax but for polling adapter that doesn't work.
        https://prod-XX.westeurope.logic.azure.com:443/workflows/<ID>/triggers/manual/paths/invoke?api-version=2016-10-01&sp=/triggers/manual/run&sv=1.0&sig=<SECRETKEY>&incrementalToken={incrementalToken}

       

      OOM Out Of Memory
      When PO reads data from an external system it should have OOM protection to not read more than X bytes. This is needed for pollig adapter, and the regular sender and receiver adapters for request and response. This setting must be per channel as the requirement is very different per scenario.
      Example decompress of a xz compressed file of size 100 MB in po can easily case OOM while a simple transport of a xz compressed file of 100 MB usually works fine.
      The sftp adapter has OOM protection on the sender adapter "maximum file size restriction" but it's missing on the http based adapters, http_aae, rest, ws_aae, soap and more adapters.
      In the webdispatcher it's possible to set max payload size but this is a global setting which isn't fine grained enough and it only works in direction sender -> po.