Skip to Content

In my documentation, I will explain and detail how install and configure SAP Hana 2.0 SP1 with Smart Data Streaming and Dynamic Tiering combined on the same server.

For my data collection, I will connect my Raspberry PI which use GrovePI for sensor to my environment, and will build a custom XS application to visualize the data captured.

For my setup, I’ll use my own lab on VMware VSphere 6.5, run SAP Hana 2.0 SP1 environment and components, Raspberry PI 3 Model B with GrovePI+ and sensors.

I will create a new environment by using vm template explain in my previous documentation.

Disclaimer: My deployment is only for test purpose; I make the security simple from a network perspective to realize this configuration and use open source software.
Important note: SAP Dynamic Tiering is no currently supported on Vmware.

In order execution

  1. Install SAP Hana Cockpit
  2. Register SAP Hana systemdb and create tenant in Cockpit
  3. Install Dynamic Tiering and add the service to the tenant
  4. Install Smart Data Streaming and add the service to the tenant
  5. Configure Raspberry PI with Streaming Lite Specialized Component
  6. Store the incoming data in Dynamic Tiering
  7. Build a custom application for my project.

Guide used:

Smart Data Streaming and Streaming Lite

  • SAP HANA Smart Data Streaming: Installation and Update Guide
  • SAP HANA Smart Data Streaming: Configuration and Administration Guide
  • SAP HANA Smart Data Streaming: Adapters Guide
  • SAP HANA Smart Data Streaming: Developer Guide

Dynamic Tiering

  • SAP HANA Dynamic Tiering: Administration Guide
  • SAP HANA Dynamic Tiering: Installation Guide

Note used:

Hana Cockpit

  • 2414668 – SAP HANA cockpit 2.0 SP 01

Link used:

 

General Picture

From a high-level architecture point of view I’ll deploy 3 vms, all registered in my internal DNS
vmckp – SAP Hana Cockpit 2.01
vmhanadb01 – Master node (Site1)
vmsdsdt01 – Worker (DT, SDS) (Site1)
iotlab – Raspberry PI with GrovePI

 

Install SAP Hana Cockpit 2.0

Starting Hana 2.0, a new dedicated instance sitting on SAP Hana Express Edition is used to administrate and manage SAP Hana instance and requires having a minimum of 16GB of Ram (Prod System)
From a deployment standpoint, SAP Hana Cockpit is a XS Advanced application, which cannot be deployed on an existing Hana instance.

I start to download the necessary software and decompress it to run the installation

The Hana Cockpit can be installed either graphical or command line, I will use the command line for my documentation

By default, the admin user created is COCKPIT_ADMIN

Once installed, we can see the urls available to log into

Let’s first log in to the cockpit manager url

On this page, this is where you need to register all your Hana resources

When you click on the bottom link “Go to SAP Hana Cockpit” this will lead you to the main Hana cockpit resource

We are done for the cockpit itself, I will proceed now with the next step.

 

Register SAP Hana systemdb and create tenant in Cockpit

In SAP Hana 2.0 SP1, the default and only installation type is to be in MCD mode, thus I have already install my systemdb and will deploy a new tenant database from the cockpit and register it as a single database.

From the cockpit manager, select “Registered Resources” and “Register Resources”

Provide all the necessary information requested as the type of database used

Once registered, the systemdb appear in the list of the registered hana system

I will create my tenant database from the SAP Hana Cockpit, because my system in now available it appears form the cockpit list. To manage it i click on resource

I select my SYSTEMDB

From the administration panel, select the “Tenant Monitoring and Admin”

And create my new tenant database

Once the tenant is up and running I register it in the cockpit

My tenant database is now installed and registered in my cockpit, i will then continue with the next part of my deployment

 

Install Dynamic Tiering and add the service to the tenant

The next step now is to install dynamic tiering on the dedicated host and add the service to the tenant database.

I have already download the Dynamic Tiering software

Once decompressed, I will run the installation but this time web based from the cockpit

From the “Add Software Location” I provide the path where my software component is stored and I keep going

From the add host and role, I just add the DT Work role to my existing system and continue

I review all my config and update the system

The installation is completed successfully

Now done I need to and assign the service to my tenant database and perform some configuration such as dedicated a specific path for es_log and es_data.

To assign the esserver service to the tenant, from the SYSTEMDB I run the command below

The next step will be to isolate the “log” and the “data” of the tenant database for DT, to do so I will first create at OS layer two specific directories which belong to my tenant DB “HB1” on my DT server

And run the two following SQL statement from the SYSTEMDB to make it active:

One done I restart my tenant and check the config from the global.ini if it’s correctly affected

Once restarted, from the cockpit for HB1 I can see now Dynamic Tiering Link on the SAP HANA Options tab

I create now my extended storage from the configure link

And check at OS layer if the data is written at the correct place

Dynamic Tiering is configured, I continue with Smart Data Streaming

Install Smart Data Streaming and add the service to the tenant

Smart Data Streaming installation for my scenario consist of several package software to downloads such as:
Note: I will not cover the SDS Studio installation portion but it’s already installed on my side

Once all downloaded, I start to run the installation of the server part from the cockpit
Note: You need to specify the Hana client source install folder if it’s not already installed

Continue and add the new role to the dedicated host

Review the configuration and update the system

The installation is completed successfully

My Streaming service is ready; I will then add it to my tenant database. I run the following statement

And we can see my Streaming link from the Tenant cockpit now

Now I need to initialize it since it not done yet.

To initialize my tenant i run the statement below from the console on the tenant database

And I can see now that streamingserver service appear as “master”

To make further validation, at OS layer 2 new folder should have been created “cluster” and “hb1”, which is what I can see

Smart Data Streaming based installation is now completed; from a summary perspective, all the server portion are completed such as:

  • Dynamic Tiering
  • Hana Cockpit
  • And Smart Data Streaming

From a VMware point of view here is the consumption on memory and cpu side per vms

I will now proceed with the Streaming Lite Specialized Component setup

 

Configure Raspberry PI with Streaming Lite Specialized Component

Streaming lite is a to-the-edge component designed to deploy streaming projects on remote gateway devices such as Raspberry PI.

For my configuration, I’m using a Raspberry PI 3 Model B in conjunction with GrovePI+, I will use 3 type of sensors from Seeed to capture data:
Air Quality Sensor v1.3 – SKU 101020078
Temp&Humi Sensor – SKU 101020011
Gas Sensor(MQ3) – SKU 101020006

Reference and website can be found from the link at the beginning of my document at “Link used”

In term of component deployment point of view, Streaming Lite will be install on my Raspberry PI and will streams output via WebSocket to Smart Data Streaming

This portion of setup will consist of:

  1. Deploy Streaming Lite component on my Raspberry PI
  2. Check Smart Data Streaming Web service enablement
  3. Build Smart Data Streaming project
  4. Build Streaming Lite project
  5. Retrieve sensor data and load them into Streaming Lite project
  6. Check live data streamed and received in Hana

 

Deploy Streaming Lite component on my Raspberry PI

The Streaming Lite component is a tar package that need to be decompress at the location of your choice, once done the “install” will be there.

To run Streaming Lite script i must set the variable environment “STEAMING_HOME” accordingly. I will do it in root user .bashrc

And I run I quick check on startup project, it’s good

 

Check Smart Data Streaming Web service as enablement

Now done, from the cockpit I will make sure the Web Service is enable for Smart Data Streaming

Note also the port used because it will be use later for the web output adapter

Now checked I can start to build my project in Hana studio for Smart Data Streaming
Note: Make sure to install the plug-in as “Administrator” for windows client machine

Build Smart Data Streaming project

The virtual scenario to construct my project is the following:
I have several workspace locations such as, store, Factory, Library … across different country that I would like to monitor the temperature, humidity, air quality and air density.

Among this location, I want to track only my IT Room location based in Mexico where 12 employees are working in this place with many computers.

To do so, I will combine my WORKSPACE table entries, which contain my location information with my sensor data received, filter only for Mexico location and store the output in specific table

My project is name “environment_sds”

Once my project build is completed, I compile it to check for no error, to do so I click on the compile button

If no error, click on the “Run Streaming Project” to test it

Before to run the test let’s have a quick view on the different windows stream to see what my incoming data will look like. I click on each component to look at them

Now I will simulate the incoming data by choosing “Manual Input” and select the input stream

I insert random value for each of my field declared in my input stream and execute

Once done I can see that 1 row has been recorded for my output adapter

My input stream is filled up with the value

My join stream has the data populated for all location

And my filtered stream with only Mexico location

Finally, for the last validation, like mention earlier I have a table to store all the captured data name “MEXFL” where my output adapter push the data into, I check if I have my data recorded

Build Streaming Lite project

The Streaming Lite project to build is very simple; it consists of having the same input stream as the SDS project and the output adapter.

Streaming Lite support only “Streaming Web Output adapter, here the detail of what should be enter for parameter:

Protocol -> websocket or rest
Destination Cluster Auth. Data -> <user/password>
SWS Server Host -> the SDS Host “not Hana”
SWS Server Port -> the communication port
Dest. Workspace project -> the workspace where the project is created
Dest. Project name -> name of the target project
Dest. Stream name -> name of the stream input in SDS to fill

Once the Streaming Lite project is done, I check the by compiling it if not error

And load it into my Raspberry PI, to do so I transfer the .ccx project file via winscp

Once transferred, I test my SL project and try to send data from my Raspberry, to do so I need to invoke the streamingproject command by specifying the .ccx location and the port to use

Before to try to inject data from my Raspberry I need to start the Stream Web Service on Smart Data Streaming node, to start it I go on the Hana cockpit and select the manage streaming node

Click on SWS and start it

Back to my Raspberry, I need to start the following string command to load my test data:

echo “IOTLAB,i,,15:04:54 PM,35,26,VERY GOOD,43” | /sap/install/bin/streamingconvert -d “,” -p iotlab:9093/environment_st | /sap/install/bin/streamingupload -p iotlab:9093/environment_st

Thus, we can see my insert successfully proceed and from a process standpoint that my connection has been initiated

Now let’s check on the Hana side if my MEXFL table has received the information, and I can see all my input based on the insert command

The load test successfully done, I will achieve the last part of configuration in my Raspberry which consist of creating a custom python script which will retrieve the sensor data.

Retrieve sensor data and load them into Streaming Lite project

For my IoT project I use a GrovePI platform on top of my raspberry for sensor, thus I don’t have any physical sensor connect to my Raspberry.

GrovePI execute python script to use the sensor and retrieve the data, so how to load those data into Streaming Lite??

Once Streaming Lite is deployed on Raspberry, it basically uses XML-RPC internally to communicate which give several options:

  • Create a custom java adapter from my hana studio and load it into my Raspberry
  • Communicate directly with the XML-RCP internal server to load my sensor input
  • Create a custom python script which encapsulate input data and execution command

I have decided to use the third option, from a sequence stand point the first thing to do is to create a special command line at os level.
In the /usr/bin I create the command called “stream” and include the following line

Now done I test my command with the syntax and check the result in Hana

So, my command is working, I will now create a test script to see if the proceed work to in python. I will voluntary put some data for input

Execute my script and check the data in Hana again

It works, after this validation I can now create the real python script which will be use to capture the sensor input and send then into Hana.

 

Check live data streamed and received in Hana

Below, here is my final script to be used to ship the sensor data captured from GrovePI into Hana

In my above script the line 25, P1 capture the sensor data and ended by a pipe command ( | ) followed by P2, line 26,  which convert and upload the data to my smart data streaming server.

In order to control the cadence of data flowing to Hana, I have decided to only read every 10 second the captured data, line 28 time.sleep(10).

And finally, to be able to compare the data read by the script vs what is incoming in Hana I have declared a print, line 27, so I can see in realtime if the data matches.

Let’s run the script now and compare

As we can see the data are read and stored to Hana in my table “MEXFL” properly, I can follow up with Dynamic Tiering part.

Store the incoming data in Dynamic Tiering

Earlier in my documentation I have installed and configured dynamic tiering, the purpose of this is to avoid having a huge amount of data loaded in memory.

To store my MEXFL table in DT I will basically right click on it and mark as “extended”
Note : I have let me sensor for a moment in order to capture enough data before to store my table in the extended storage.

Moving table in extend storage is a very simple move, however you will need to take some consideration such as the volume of data to move and the disk write speed performance.

For my lab I’m using SAS storage with 10k speed and also I’m controlling the frequency of the data sending (every 10 sec), for a real life case scenario consider to read the following document for guidance and best practice recommendation

I am done with the DT part now, so I will go to the next part of my documentation with the creation of my custom application.

Build a custom application for my project

To expose and visualize my data I will create a basic XS application, SAP HANA XS can be used to expose data from Tables, Views and Modeling Views to UI layer.
There are major 2 ways to expose data the data to clients – OData and XSJS, in my case I will use the OData option.

I will briefly explain my step but won’t go deep since it’s not the main goal of this documentation, how do I proceed? here are the step

I start to build the logic (calculation views), for my setup I have created one calc view per application which will be used for my xsodata services

Once my calc view are all set, I create my xsapp and 3 folders such as:

  • “apps” – where I store my xswidget that will contain my dynamic application
  • “catalog” – where I store my xsappsite, which contain the catalog for application
  • “odata” – where i store my xsodata service used for each individual dynamic application

Once my build completed this is what my application look like

My setup is completed for this part, in my next blog will explain how to configure HSR for this particular setup with Dynamic Tiering and Smart Data Streaming.

To report this post you need to login first.

4 Comments

You must be Logged on to comment or reply to a post.

    1. Williams Ruter Post author

      Thanks Chandan, more things coming up, i have other blog on my ex-sap employee account. You can check them also, look on the left side of my profile in the section “elsewhere”

      More stuff coming up very shortly !!! it takes time to put all this stuff together 😉

      Williams

       

      (1) 
    1. Williams Ruter Post author

      Hey Bilem, thanks for the comment, more stuff coming stay tune …

      You can also check out my other blog already publish if you look at the “elsewhere” section of my profile if interested.

      Williams

      (0) 

Leave a Reply