This blog post is written in collaboration with the SAP HANA Database & Analytics, Cross Product Management team - thanks to my colleague wei.han7 for her help in reviewing this post
Using the
SAP Data Intelligence Modeler, we can create
pipelines (graphs) to work with data from a number of sources. One of the first things you may want to try when creating your first pipeline is retrieving data via an API
In this brief blog post, we'll cover the basics of using the
OpenAPI Client operator to retrieve data from a
Qualtrics API - List Surveys. Once you understand the basics of working with the
OpenAPI Client operator, you should be equipped to retrieve data from other API sources
Our API Request
By default, the Qualtrics API Reference presents a sample API Request for Shell/cURL (the
x value in
X-API-TOKEN is a stand-in for your individual
Qualtrics API Token)
Sample API Request
At time of writing, the
OpenAPI Client operator uses the
OpenAPI 2.0 specification (formerly known as Swagger 2.0 specification). In this blog post, we will demonstrate how to map our
Qualtrics API Request to this specification, however any further exploration of the OpenAPI specification is considered out of scope
Next we will map the
Sample API request to the fields of our
OpenAPI Connection and Client operator. A visual breakdown is included below
Fields in the Connection are marked in Green, Operator fields marked in Blue
Creating our Connection
Creating the
DI Connection for Qualtrics allows for centralized management of the connection if, for example you rely on more than one API which uses the same connection details
From the
Data Intelligence Launchpad, select
Connection Management then click on the
Create Connection button
Click to create a new Connection
For the
Connection Type, we want to choose
OpenAPI
Host is used to specify the host (name or IP) serving the API. This does not include the scheme or sub-path, but
may include the port. For this example, our host is
ca1.qualtrics.com
Protocol (Scheme) specifies the transfer protocol of the API. In our case, the protocol used is
https
Select
ApiKey as our
Authentication Type, and fill out the values for
API Key Name (this API expects 'X-API-TOKEN') and API Key Type (this can be either 'Header' or 'Query'. As we can see from our sample request, the API Key should be sent in the 'Header')
Creating the connection in Connection Management
We can retrieve our
API Key Value from
Qualtrics. We do this by navigating to
Account Settings -> Qualtrics IDs ->IDs
Retrieving our Qualtrics API Token
The value in
API Token is the final value we need for our
DI Connection. We copy it to API Key Value field, click
Create and return to the
Modeler.
Creating our Pipeline
First, we start by placing an
OpenAPI Client operator on a new
graph
OpenAPI Client operator can be found under Connectivity
First, we will select the
OpenAPI Client operator. In our case, the values required for
Method and
Produces are correct by default - we are making a
GET request, and the result is expected in the
application/json format. Neither of these values need to be changed
Base Path specifies the base path where the API is served, which is relative to the host. It must begin with a leading slash (/). We want to enter
/API/v3/surveys. We then want to select the
Connection we just created
Setting the Base Path and Connection for our OpenAPI Client Operator
Completing our Graph
At this point, all of the necessary
configuration of the
OpenAPI Client operator has been completed. The
operator will make a request when it receives input to its
input port. Without the Workflow Trigger operator (or another compatible operator), the openAPI Client will not run
In order for our example to trigger the
API Call exactly once, we will place a
Workflow Trigger operator into the graph, and connect its
output port to the
input port of our
OpenAPI Client operator
In order to read the
API response, we connect the
output port of our
OpenAPI Client operator to a
Wiretap operator. Our graph is now ready to be tested
Our finished graph
Save and
run your graph, then select the
Wiretap operator and click on the
Open UI symbol to check the output
Our Wiretap Output shows that our request is successful
In this blog post, we've covered how we can use an
OpenAPI Client operator to make an API request, and verify the response data in a
Wiretap operator. We can store these results in a table in HANA, and use them in combination with other APIs (for example, retrieving responses for individual surveys) for modeling or analytics
This blog post may naturally lead to follow up questions, such as "What if I need to pass in additional parameters", or "What if I want to dynamically change the values of parameters?". I hope to cover these questions in future blog posts
I hope this blog post has been useful, and I welcome any comments or questions below
Edited March 2022 - Updated the blog post to reflect the removal of the OpenAPI Client's manual connection mode