Use XSJS outbound connectivity to search tweets

[Former Member]

Intro

XSJS outbound connectivity was introduced in SAP HANA SPS06. It is really a cool feature in SAP HANA XS. With XSJS outbound connectivity, we can directly make HTTP/HTTPS request in our SAP HANA native app. I love social media and this feature makes it possible to connect social media (e.g. Twitter, Facebook) API with SAP HANA and do some interesting analysis. In this blog I want to share with you how to use XSJS outbound connectivity to search tweets.

Motivation

Last year I had a mini-fellowship in Palo Alto and it was my first time in the US. I wanted to watch a movie one weekend but it's hard for me to pick one. So I decided to use SAP HANA to build a smart app which would tell me the rating of movies. I made it and selected the movie with highest rating. The main idea is first crawl tweets and insert into SAP HANA, then use the native text analysis to calculate the sentiment rating. However, when I was building this app, SAP HANA is still SPS05. Without XSJS outbound connectivity, I have to use Twitter4J to connect Twitter API. If only I could use XSJS outbound connectivity at that time! Never mind. It's time to rebuild the smart app and replacing Twitter4J part with XSJS outbound connectivity is my first step.

Now let's do it.

Prerequisites

1. A running SAP HANA system, at least SPS06. I am using SAP HANA SPS08 Rev. 80.

2. A Twitter account. If you don't have, sign up here.

Steps

1. Learn how to communicate with Twitter API

In this step, we need to find what APIs will be used and how to communicate with Twitter API (authentication & authorization stuff). First you can find all Rest APIs here. We want to search tweets, so we will use GET search/tweets | Twitter Developers It is very clear and you can find the URL, parameters and an example request.

But how can we call this API? You can find the answer in this doc. Since we just need to search tweets, we can use the app-only authentication.There you can find a very detailed example with three steps. That's exactly what we need. There is one thing need to be mentioned, also you can find it here "As with all API v1.1 methods, HTTPS is always required."

2. Use Postman to simulate calling the API

Since now we know how to communicate with Twitter API, we can first test it with Postman. I like to test before I really do something. There is also a lot of other tools similar with Postman, please use your favorite. The steps are described clearly in app-only authentication. I will not describe them again. Here I just summarized my steps with some screenshots.

a. Encode API key and secret.

First you need to create an app if you don't have one. You can find <API key> and <API secret> under "API Keys" tab in you app. I've already regenerated API key and secret, so the key and secret in the following pic do not work now.

Then encode <API key>:<API secret> to Base64 format. For example, you can use Base64 Decode and Encode - Online

b. Obtain a bearer token

You need to sign out your Twitter account in Chrome; otherwise you will not get the bearer token. Instead you will get this message "403 Forbidden: The server understood the request, but is refusing to fulfill it." I've already invalidated the bearer token in the following pic.

c. Test GET search/tweets | Twitter Developers with the bearer token

We managed to search tweets with hashtag #SAPHANA and got the results. To keep simplicity, we just use parameter "q" which stands for query.

3. Set up your SAP HANA to use HTTPS

Till now, we have successfully called one Twitter API using Postman. So why not use XSJS outbound connectivity? Let's start! As with all API v1.1 methods, HTTPS is always required. So, the first thing we need to do is set up our SAP HANA to use HTTPS which is not configured by default. You can follow this detailed blog by Kai-Christoph to finish this step. When you finish it, you should be able to do the following; Otherwise you do not finish this step.

a. Visit https://<hostname or IP>:43<instance number>/sap/hana/xs/admin/ successfully

b. When you switch to "Trust Manager" tab, there is no such error "No valid SAP crypto configuration"

4. Create trust store of Twitter API

In this step, we need to create trust store of Twitter API. Again, you can follow this detailed blog by Kai-Christoph to finish this step. There is only one thing you need to change. That is visit https://api.twitter.com/ instead of https://api.github.com/ in the blog.

5. Use XSJS outbound connectivity to search tweets

We finally come to this step. Since we have prepared everything in the previous steps. It is easy for us in this step. We just need to do the following. I did it in SAP HANA Studio. Of course you can also do it in Web IDE. Here is my project hierarchy. It is very simple.

a. Create a XS project

b. Create .xsapp, .xsaccess and "services" folder

c. Create twitterApi.xshttpdest, edit, save and activate

description = "twitter api";

host = "api.twitter.com";

port = 443;

pathPrefix = "/1.1";

useProxy = true;

proxyHost = "proxy.pal.sap.corp";

proxyPort = 8080;

authType = none;

useSSL = true;

timeout = 0;

d. Edit trust store in HTTP destination (in red box) and save

e. Create search.xsjs and edit. From Application-only authentication | Twitter Developers, we can learn that "Note that one bearer token is valid for an application at a time. Issuing another request with the same credentials to /oauth2/token will return the same token until it is invalidated." So, we do not need to obtain the bearer token each time which means we can directly use bearer token in our code. I've already invalidated the bearer token in the following code.

var destination = $.net.http.readDestination("searchTweets.services", "twitterApi");

var client = new $.net.http.Client();

var request = new $.net.http.Request($.net.http.GET, "/search/tweets.json?q=%23SAPHANA");

request.headers.set('Authorization', 'Bearer AAAAAAAAAAAAAAAAAAAAADdMZgAAAAAADme2QQk3csQXnGCeepM7Swvf6PI%3DJZNlbYr3YkcDsS0xCgeRgmzJW5Cjk8cvI4ESXECzVKTYI3bNw5');

var response = client.request(request, destination).getResponse();

$.response.status = response.status;

$.response.contentType = response.contentType;

$.response.setBody(response.body.asString());

f. Save and activate all files

6. Test it

Now let's test our XSJS outbound connectivity. Called successfully! There is a tweet text which contains "The only limitation is our imagination!"

Conclusion

So far, we have successfully used XSJS outbound connectivity to search tweets. However, yet we neither inserted tweets into SAP HANA nor made some analysis. I'll do this in my smart app and share with you later. In addition, there is also a lot of other Twitter APIs which you can call with XSJS outbound connectivity.

Hope you enjoyed reading my blog. :smile: