My Experiments with SAP Streaming Analytics on Google Cloud

PS: Demo Video at the end.

In my previous blog after facing so many issues finally I was able to deploy SAP HANA express edition on Google Cloud platform with help of SAP Community support.  After exploring a bit I was searching for the next adventure and this is when SAP HANA Streaming Analytics came into picture.

I was thinking of using some real time sensor data. For people like me who are yet to get their hands on Rasperry Pi or Arduino etc., for us our laptop and mobiles are our sensors our IoT devices:). So this time i thought of using my laptop as my streaming device.

What we are trying to achieve.

I wanted to monitor my laptop RAM usage if it goes beyond a certain level I shall receive an SMS considering I am not around the laptop.

Technology Stack used

• Python to read RAM usage of laptop
• SAP Streaming Analytics API for posting the laptop data
• SAP Streaming Analytics Modeler for design, receiving and triggering of Alarms.
• SAP Streaming HTTP JSON adapter
• PHP script to be called from SAP Streaming alarms
• SAP API Hub Proximus Enco API to send alerts

How we did it?

At high level the Laptop was sending data via python script to SAP Streaming Analytics API. The Streaming Analytics server analysed the data and then raised alarms based on memory usage, in our case we have kept anything above 50% usage of RAM shall raise an alarm

Set up

In order to set up SAP HANA Streaming Analytics on google cloud I followed step 7 of this wonderful blog by Lucia Subatin. I needed to make a small adjustment in modifying the hosts file on Google cloud platform. The tutorial mentions to modify 127.0.0.1 but instead we have to modify the entry which is maintained against HXEHOST.LOCALDOMAIN HXEHOST in my case it was 127.0.0.2 . Secondly the file name is HSA.TZ.

Developing – Python Part

I created a local python script using PSUTIL library which helps in reading the RAM usage of the laptop. After this I called the SAP Streaming Analytics API to post the readings. As can be seen in the code, the first call is to retrieve the token and next call is to post the data.

import psutil
import time
import requests
import json
import ast
# Get Authorization token
authorizationURL = 'http://hxehost:39045/1/authorization'
headersAuthorization = {
    "Authorization": "Basic <base64 username/password>"    
}
payloadAuthorization = [
{ "privilege":"write", "resourceType":"stream", "resource":"default/zram2/MACHINEDATA"}
]
responseAuthorization = requests.post(authorizationURL,data=json.dumps(payloadAuthorization), headers=headersAuthorization)
r = responseAuthorization
r1 = json.loads(json.dumps(r.text))
r2 = r1.split(':')
r3 = r2[1].split("}")

#Post Data
postDataURL = 'http://hxehost:39045/1/workspaces/default/projects/zram2/streams/MACHINEDATA'
headersPostData = {
"authorization":"SWS-Token \"sws-token\"=" + r3[0]	
}
#print(headersPostData)

count = 0
while count < 5:
	count += 1
	p = psutil.virtual_memory()
	print p.percent
	time.sleep(10)
	payloadPostData = [
	{"ESP_OPS":"i", "MACHINEID":"1A", "EVENT_TIME": "2018-05-14T04:45:36.123", "EVENT_NAME":"TEMP"
	, "EVENT_DESCRIPTION":"RAM Usage", "EVENT_VALUE":str(p.percent)}
    ]

	responsePost = requests.post(postDataURL,data=json.dumps(payloadPostData), headers=headersPostData) 
#	print(payloadPostData)
	print(responsePost)
	

Developing – SAP Streaming Analytics Trigger Alarm & Trigger HTTP call part

I have reused the SAP tutorial Get started with SAP HANA Smart Data streaming. This tutorial although monitor temperature, I made the following tweaks to adapt to my RAM usage scenario.

• Added HTTP Client JSON output adaptor to call the PHP script which will consume SAP API Hub SMS API.

• Renamed the labels from temperature to RAM
• Removed the unwanted fields from the window.

 Developing – Alert SMS part

For this I have reused the API which we have used in my previous blog of approvals via SMS and adapted as per the need. The sample code is added below.

<?php
$request = file_get_contents('php://input');
$json= json_decode($request);
var_dump($json);
// Send SMS API$context = array();

$ch1 = curl_init('https://sandbox.api.sap.com/proximusenco/sms/outboundmessages');
$request_headers = array();
$request_headers[] = 'Content-Type: application/json';
$request_headers[] = 'Accept: application/json';
$request_headers[] = 'APIKey: <API Key>';
curl_setopt($ch1, CURLOPT_HTTPHEADER, $request_headers );
curl_setopt($ch1, CURLOPT_CUSTOMREQUEST, 'POST');
curl_setopt($ch1, CURLOPT_VERBOSE, 1);
curl_setopt($ch1, CURLOPT_HEADER, 1);
$context["message"] = "High RAM usage detected on Laptop";
$context["binary"] = false;
$context["destinations"] = ['+9111111111'];
var_dump($context);
curl_setopt($ch1, CURLOPT_POSTFIELDS,json_encode($context)           );
$resp = curl_exec($ch1);
curl_close($ch1);
?>

Live Demo & Github Link for source code.

Please find below the demo video and here is the github link.

Challenges Faced & Resolution

• Entry to be maintained in Eclipse to connect to google cloud – This is where I went crazy, I was trying to connect remotely via eclipse to SAP HANA Express edition on Google Cloud. It never worked. I was putting the host name as http://hxehost where as it all it need was hxehost. Thanks to Lucia in resolving this issue via my question posted on SCN.

• Which port to be used for posting data to SAP Streaming API – This was finally resolved by checking Streaming Configuration tab where the port number is maintained in my case 39045. As mentioned by Robert Waywell in the comments section rather than using 39045 we shall use the port web services port for cloud based VM deployments. This port can can be found under web services provider tab. This actually resolve my testing from Eclipse issue.

• SAP Streaming Analytics Installation – You need to be patient with the installation, at one point I thought nothing is going on for last 5 minutes let me redo again. This was the mistake where I cancelled in between and for some reason was not able to install again. I have to redo my whole SAP HANA Express set up.

Open Question till now

• I am able to compile my streaming project in eclipse studio but not able to test it. I have posted an open question on this. I have the workaround where I go manually in to Manager Streaming Projects and upload the project.

 What is next?

• It is still a basic stuff which is done, I am thinking can we use HANA Machine learning stuff to make it more powerful, so that is on the radar.
• A Native app for the same to get more exposure HANA XSA