In my documentation, I will explain and detail how install and configure SAP Hana 2.0 SP1 with Smart Data Streaming and Dynamic Tiering combined on the same server.
For my data collection, I will connect my Raspberry PI which use GrovePI for sensor to my environment, and will build a custom XS application to visualize the data captured.
For my setup, I’ll use my own lab on VMware VSphere 6.5, run SAP Hana 2.0 SP1 environment and components, Raspberry PI 3 Model B with GrovePI+ and sensors.
I will create a new environment by using vm template explain in my previous documentation.
Disclaimer: My deployment is only for test purpose; I make the security simple from a network perspective to realize this configuration and use open source software.
Important note: SAP Dynamic Tiering is no currently supported on Vmware.
Smart Data Streaming and Streaming Lite
Dynamic Tiering
Hana Cockpit
From a high-level architecture point of view I’ll deploy 3 vms, all registered in my internal DNS
vmckp – SAP Hana Cockpit 2.01
vmhanadb01 – Master node (Site1)
vmsdsdt01 – Worker (DT, SDS) (Site1)
iotlab - Raspberry PI with GrovePI
Starting Hana 2.0, a new dedicated instance sitting on SAP Hana Express Edition is used to administrate and manage SAP Hana instance and requires having a minimum of 16GB of Ram (Prod System)
From a deployment standpoint, SAP Hana Cockpit is a XS Advanced application, which cannot be deployed on an existing Hana instance.
I start to download the necessary software and decompress it to run the installation
The Hana Cockpit can be installed either graphical or command line, I will use the command line for my documentation
By default, the admin user created is COCKPIT_ADMIN
Once installed, we can see the urls available to log into
Let’s first log in to the cockpit manager url
On this page, this is where you need to register all your Hana resources
When you click on the bottom link “Go to SAP Hana Cockpit” this will lead you to the main Hana cockpit resource
We are done for the cockpit itself, I will proceed now with the next step.
In SAP Hana 2.0 SP1, the default and only installation type is to be in MCD mode, thus I have already install my systemdb and will deploy a new tenant database from the cockpit and register it as a single database.
From the cockpit manager, select “Registered Resources” and “Register Resources”
Provide all the necessary information requested as the type of database used
Once registered, the systemdb appear in the list of the registered hana system
I will create my tenant database from the SAP Hana Cockpit, because my system in now available it appears form the cockpit list. To manage it i click on resource
I select my SYSTEMDB
From the administration panel, select the “Tenant Monitoring and Admin”
And create my new tenant database
Once the tenant is up and running I register it in the cockpit
My tenant database is now installed and registered in my cockpit, i will then continue with the next part of my deployment
The next step now is to install dynamic tiering on the dedicated host and add the service to the tenant database.
I have already download the Dynamic Tiering software
Once decompressed, I will run the installation but this time web based from the cockpit
From the “Add Software Location” I provide the path where my software component is stored and I keep going
From the add host and role, I just add the DT Work role to my existing system and continue
I review all my config and update the system
The installation is completed successfully
Now done I need to and assign the service to my tenant database and perform some configuration such as dedicated a specific path for es_log and es_data.
To assign the esserver service to the tenant, from the SYSTEMDB I run the command below
The next step will be to isolate the “log” and the “data” of the tenant database for DT, to do so I will first create at OS layer two specific directories which belong to my tenant DB “HB1” on my DT server
And run the two following SQL statement from the SYSTEMDB to make it active:
One done I restart my tenant and check the config from the global.ini if it’s correctly affected
Once restarted, from the cockpit for HB1 I can see now Dynamic Tiering Link on the SAP HANA Options tab
I create now my extended storage from the configure link
And check at OS layer if the data is written at the correct place
Dynamic Tiering is configured, I continue with Smart Data Streaming
Smart Data Streaming installation for my scenario consist of several package software to downloads such as:
Note: I will not cover the SDS Studio installation portion but it’s already installed on my side
Once all downloaded, I start to run the installation of the server part from the cockpit
Note: You need to specify the Hana client source install folder if it’s not already installed
Continue and add the new role to the dedicated host
Review the configuration and update the system
The installation is completed successfully
My Streaming service is ready; I will then add it to my tenant database. I run the following statement
And we can see my Streaming link from the Tenant cockpit now
Now I need to initialize it since it not done yet.
To initialize my tenant i run the statement below from the console on the tenant database
And I can see now that streamingserver service appear as “master”
To make further validation, at OS layer 2 new folder should have been created “cluster” and “hb1”, which is what I can see
Smart Data Streaming based installation is now completed; from a summary perspective, all the server portion are completed such as:
From a VMware point of view here is the consumption on memory and cpu side per vms
I will now proceed with the Streaming Lite Specialized Component setup
Streaming lite is a to-the-edge component designed to deploy streaming projects on remote gateway devices such as Raspberry PI.
For my configuration, I’m using a Raspberry PI 3 Model B in conjunction with GrovePI+, I will use 3 type of sensors from Seeed to capture data:
Air Quality Sensor v1.3 – SKU 101020078
Temp&Humi Sensor – SKU 101020011
Gas Sensor(MQ3) – SKU 101020006
Reference and website can be found from the link at the beginning of my document at “Link used”
In term of component deployment point of view, Streaming Lite will be install on my Raspberry PI and will streams output via WebSocket to Smart Data Streaming
This portion of setup will consist of:
The Streaming Lite component is a tar package that need to be decompress at the location of your choice, once done the “install” will be there.
To run Streaming Lite script i must set the variable environment “STEAMING_HOME” accordingly. I will do it in root user .bashrc
And I run I quick check on startup project, it’s good
Now done, from the cockpit I will make sure the Web Service is enable for Smart Data Streaming
Note also the port used because it will be use later for the web output adapter
Now checked I can start to build my project in Hana studio for Smart Data Streaming
Note: Make sure to install the plug-in as “Administrator” for windows client machine
The virtual scenario to construct my project is the following:
I have several workspace locations such as, store, Factory, Library … across different country that I would like to monitor the temperature, humidity, air quality and air density.
Among this location, I want to track only my IT Room location based in Mexico where 12 employees are working in this place with many computers.
To do so, I will combine my WORKSPACE table entries, which contain my location information with my sensor data received, filter only for Mexico location and store the output in specific table
My project is name “environment_sds”
Once my project build is completed, I compile it to check for no error, to do so I click on the compile button
If no error, click on the “Run Streaming Project” to test it
Before to run the test let’s have a quick view on the different windows stream to see what my incoming data will look like. I click on each component to look at them
Now I will simulate the incoming data by choosing “Manual Input” and select the input stream
I insert random value for each of my field declared in my input stream and execute
Once done I can see that 1 row has been recorded for my output adapter
My input stream is filled up with the value
My join stream has the data populated for all location
And my filtered stream with only Mexico location
Finally, for the last validation, like mention earlier I have a table to store all the captured data name “MEXFL” where my output adapter push the data into, I check if I have my data recorded
The Streaming Lite project to build is very simple; it consists of having the same input stream as the SDS project and the output adapter.
Streaming Lite support only “Streaming Web Output adapter, here the detail of what should be enter for parameter:
Protocol -> websocket or rest
Destination Cluster Auth. Data -> <user/password>
SWS Server Host -> the SDS Host “not Hana”
SWS Server Port -> the communication port
Dest. Workspace project -> the workspace where the project is created
Dest. Project name -> name of the target project
Dest. Stream name -> name of the stream input in SDS to fill
Once the Streaming Lite project is done, I check the by compiling it if not error
And load it into my Raspberry PI, to do so I transfer the .ccx project file via winscp
Once transferred, I test my SL project and try to send data from my Raspberry, to do so I need to invoke the streamingproject command by specifying the .ccx location and the port to use
Before to try to inject data from my Raspberry I need to start the Stream Web Service on Smart Data Streaming node, to start it I go on the Hana cockpit and select the manage streaming node
Click on SWS and start it
Back to my Raspberry, I need to start the following string command to load my test data:
echo "IOTLAB,i,,15:04:54 PM,35,26,VERY GOOD,43" | /sap/install/bin/streamingconvert -d "," -p iotlab:9093/environment_st | /sap/install/bin/streamingupload -p iotlab:9093/environment_st
Thus, we can see my insert successfully proceed and from a process standpoint that my connection has been initiated
Now let’s check on the Hana side if my MEXFL table has received the information, and I can see all my input based on the insert command
The load test successfully done, I will achieve the last part of configuration in my Raspberry which consist of creating a custom python script which will retrieve the sensor data.
For my IoT project I use a GrovePI platform on top of my raspberry for sensor, thus I don’t have any physical sensor connect to my Raspberry.
GrovePI execute python script to use the sensor and retrieve the data, so how to load those data into Streaming Lite??
Once Streaming Lite is deployed on Raspberry, it basically uses XML-RPC internally to communicate which give several options:
I have decided to use the third option, from a sequence stand point the first thing to do is to create a special command line at os level.
In the /usr/bin I create the command called “stream” and include the following line
Now done I test my command with the syntax and check the result in Hana
So, my command is working, I will now create a test script to see if the proceed work to in python. I will voluntary put some data for input
Execute my script and check the data in Hana again
It works, after this validation I can now create the real python script which will be use to capture the sensor input and send then into Hana.
Below, here is my final script to be used to ship the sensor data captured from GrovePI into Hana
In my above script the line 25, P1 capture the sensor data and ended by a pipe command ( | ) followed by P2, line 26, which convert and upload the data to my smart data streaming server.
In order to control the cadence of data flowing to Hana, I have decided to only read every 10 second the captured data, line 28 time.sleep(10).
And finally, to be able to compare the data read by the script vs what is incoming in Hana I have declared a print, line 27, so I can see in realtime if the data matches.
Let’s run the script now and compare
As we can see the data are read and stored to Hana in my table “MEXFL” properly, I can follow up with Dynamic Tiering part.
Earlier in my documentation I have installed and configured dynamic tiering, the purpose of this is to avoid having a huge amount of data loaded in memory.
To store my MEXFL table in DT I will basically right click on it and mark as “extended”
Note : I have let me sensor for a moment in order to capture enough data before to store my table in the extended storage.
Moving table in extend storage is a very simple move, however you will need to take some consideration such as the volume of data to move and the disk write speed performance.
For my lab I’m using SAS storage with 10k speed and also I’m controlling the frequency of the data sending (every 10 sec), for a real life case scenario consider to read the following document for guidance and best practice recommendation
I am done with the DT part now, so I will go to the next part of my documentation with the creation of my custom application.
To expose and visualize my data I will create a basic XS application, SAP HANA XS can be used to expose data from Tables, Views and Modeling Views to UI layer.
There are major 2 ways to expose data the data to clients – OData and XSJS, in my case I will use the OData option.
I will briefly explain my step but won’t go deep since it’s not the main goal of this documentation, how do I proceed? here are the step
I start to build the logic (calculation views), for my setup I have created one calc view per application which will be used for my xsodata services
Once my calc view are all set, I create my xsapp and 3 folders such as:
Once my build completed this is what my application look like
My setup is completed for this part, in my next blog will explain how to configure HSR for this particular setup with Dynamic Tiering and Smart Data Streaming.
For my data collection, I will connect my Raspberry PI which use GrovePI for sensor to my environment, and will build a custom XS application to visualize the data captured.
For my setup, I’ll use my own lab on VMware VSphere 6.5, run SAP Hana 2.0 SP1 environment and components, Raspberry PI 3 Model B with GrovePI+ and sensors.
I will create a new environment by using vm template explain in my previous documentation.
Disclaimer: My deployment is only for test purpose; I make the security simple from a network perspective to realize this configuration and use open source software.
Important note: SAP Dynamic Tiering is no currently supported on Vmware.
In order execution
- Install SAP Hana Cockpit
- Register SAP Hana systemdb and create tenant in Cockpit
- Install Dynamic Tiering and add the service to the tenant
- Install Smart Data Streaming and add the service to the tenant
- Configure Raspberry PI with Streaming Lite Specialized Component
- Store the incoming data in Dynamic Tiering
- Build a custom application for my project.
Guide used:
Smart Data Streaming and Streaming Lite
- SAP HANA Smart Data Streaming: Installation and Update Guide
- SAP HANA Smart Data Streaming: Configuration and Administration Guide
- SAP HANA Smart Data Streaming: Adapters Guide
- SAP HANA Smart Data Streaming: Developer Guide
Dynamic Tiering
- SAP HANA Dynamic Tiering: Administration Guide
- SAP HANA Dynamic Tiering: Installation Guide
Note used:
Hana Cockpit
- 2414668 - SAP HANA cockpit 2.0 SP 01
Link used:
General Picture
From a high-level architecture point of view I’ll deploy 3 vms, all registered in my internal DNS
vmckp – SAP Hana Cockpit 2.01
vmhanadb01 – Master node (Site1)
vmsdsdt01 – Worker (DT, SDS) (Site1)
iotlab - Raspberry PI with GrovePI
Install SAP Hana Cockpit 2.0
Starting Hana 2.0, a new dedicated instance sitting on SAP Hana Express Edition is used to administrate and manage SAP Hana instance and requires having a minimum of 16GB of Ram (Prod System)
From a deployment standpoint, SAP Hana Cockpit is a XS Advanced application, which cannot be deployed on an existing Hana instance.
I start to download the necessary software and decompress it to run the installation
The Hana Cockpit can be installed either graphical or command line, I will use the command line for my documentation
By default, the admin user created is COCKPIT_ADMIN
Once installed, we can see the urls available to log into
Let’s first log in to the cockpit manager url
On this page, this is where you need to register all your Hana resources
When you click on the bottom link “Go to SAP Hana Cockpit” this will lead you to the main Hana cockpit resource
We are done for the cockpit itself, I will proceed now with the next step.
Register SAP Hana systemdb and create tenant in Cockpit
In SAP Hana 2.0 SP1, the default and only installation type is to be in MCD mode, thus I have already install my systemdb and will deploy a new tenant database from the cockpit and register it as a single database.
From the cockpit manager, select “Registered Resources” and “Register Resources”
Provide all the necessary information requested as the type of database used
Once registered, the systemdb appear in the list of the registered hana system
I will create my tenant database from the SAP Hana Cockpit, because my system in now available it appears form the cockpit list. To manage it i click on resource
I select my SYSTEMDB
From the administration panel, select the “Tenant Monitoring and Admin”
And create my new tenant database
Once the tenant is up and running I register it in the cockpit
My tenant database is now installed and registered in my cockpit, i will then continue with the next part of my deployment
Install Dynamic Tiering and add the service to the tenant
The next step now is to install dynamic tiering on the dedicated host and add the service to the tenant database.
I have already download the Dynamic Tiering software
Once decompressed, I will run the installation but this time web based from the cockpit
From the “Add Software Location” I provide the path where my software component is stored and I keep going
From the add host and role, I just add the DT Work role to my existing system and continue
I review all my config and update the system
The installation is completed successfully
Now done I need to and assign the service to my tenant database and perform some configuration such as dedicated a specific path for es_log and es_data.
To assign the esserver service to the tenant, from the SYSTEMDB I run the command below
The next step will be to isolate the “log” and the “data” of the tenant database for DT, to do so I will first create at OS layer two specific directories which belong to my tenant DB “HB1” on my DT server
And run the two following SQL statement from the SYSTEMDB to make it active:
One done I restart my tenant and check the config from the global.ini if it’s correctly affected
Once restarted, from the cockpit for HB1 I can see now Dynamic Tiering Link on the SAP HANA Options tab
I create now my extended storage from the configure link
And check at OS layer if the data is written at the correct place
Dynamic Tiering is configured, I continue with Smart Data Streaming
Install Smart Data Streaming and add the service to the tenant
Smart Data Streaming installation for my scenario consist of several package software to downloads such as:
Note: I will not cover the SDS Studio installation portion but it’s already installed on my side
Once all downloaded, I start to run the installation of the server part from the cockpit
Note: You need to specify the Hana client source install folder if it’s not already installed
Continue and add the new role to the dedicated host
Review the configuration and update the system
The installation is completed successfully
My Streaming service is ready; I will then add it to my tenant database. I run the following statement
And we can see my Streaming link from the Tenant cockpit now
Now I need to initialize it since it not done yet.
To initialize my tenant i run the statement below from the console on the tenant database
And I can see now that streamingserver service appear as “master”
To make further validation, at OS layer 2 new folder should have been created “cluster” and “hb1”, which is what I can see
Smart Data Streaming based installation is now completed; from a summary perspective, all the server portion are completed such as:
- Dynamic Tiering
- Hana Cockpit
- And Smart Data Streaming
From a VMware point of view here is the consumption on memory and cpu side per vms
I will now proceed with the Streaming Lite Specialized Component setup
Configure Raspberry PI with Streaming Lite Specialized Component
Streaming lite is a to-the-edge component designed to deploy streaming projects on remote gateway devices such as Raspberry PI.
For my configuration, I’m using a Raspberry PI 3 Model B in conjunction with GrovePI+, I will use 3 type of sensors from Seeed to capture data:
Air Quality Sensor v1.3 – SKU 101020078
Temp&Humi Sensor – SKU 101020011
Gas Sensor(MQ3) – SKU 101020006
Reference and website can be found from the link at the beginning of my document at “Link used”
In term of component deployment point of view, Streaming Lite will be install on my Raspberry PI and will streams output via WebSocket to Smart Data Streaming
This portion of setup will consist of:
- Deploy Streaming Lite component on my Raspberry PI
- Check Smart Data Streaming Web service enablement
- Build Smart Data Streaming project
- Build Streaming Lite project
- Retrieve sensor data and load them into Streaming Lite project
- Check live data streamed and received in Hana
Deploy Streaming Lite component on my Raspberry PI
The Streaming Lite component is a tar package that need to be decompress at the location of your choice, once done the “install” will be there.
To run Streaming Lite script i must set the variable environment “STEAMING_HOME” accordingly. I will do it in root user .bashrc
And I run I quick check on startup project, it’s good
Check Smart Data Streaming Web service as enablement
Now done, from the cockpit I will make sure the Web Service is enable for Smart Data Streaming
Note also the port used because it will be use later for the web output adapter
Now checked I can start to build my project in Hana studio for Smart Data Streaming
Note: Make sure to install the plug-in as “Administrator” for windows client machine
Build Smart Data Streaming project
The virtual scenario to construct my project is the following:
I have several workspace locations such as, store, Factory, Library … across different country that I would like to monitor the temperature, humidity, air quality and air density.
Among this location, I want to track only my IT Room location based in Mexico where 12 employees are working in this place with many computers.
To do so, I will combine my WORKSPACE table entries, which contain my location information with my sensor data received, filter only for Mexico location and store the output in specific table
My project is name “environment_sds”
Once my project build is completed, I compile it to check for no error, to do so I click on the compile button
If no error, click on the “Run Streaming Project” to test it
Before to run the test let’s have a quick view on the different windows stream to see what my incoming data will look like. I click on each component to look at them
Now I will simulate the incoming data by choosing “Manual Input” and select the input stream
I insert random value for each of my field declared in my input stream and execute
Once done I can see that 1 row has been recorded for my output adapter
My input stream is filled up with the value
My join stream has the data populated for all location
And my filtered stream with only Mexico location
Finally, for the last validation, like mention earlier I have a table to store all the captured data name “MEXFL” where my output adapter push the data into, I check if I have my data recorded
Build Streaming Lite project
The Streaming Lite project to build is very simple; it consists of having the same input stream as the SDS project and the output adapter.
Streaming Lite support only “Streaming Web Output adapter, here the detail of what should be enter for parameter:
Protocol -> websocket or rest
Destination Cluster Auth. Data -> <user/password>
SWS Server Host -> the SDS Host “not Hana”
SWS Server Port -> the communication port
Dest. Workspace project -> the workspace where the project is created
Dest. Project name -> name of the target project
Dest. Stream name -> name of the stream input in SDS to fill
Once the Streaming Lite project is done, I check the by compiling it if not error
And load it into my Raspberry PI, to do so I transfer the .ccx project file via winscp
Once transferred, I test my SL project and try to send data from my Raspberry, to do so I need to invoke the streamingproject command by specifying the .ccx location and the port to use
Before to try to inject data from my Raspberry I need to start the Stream Web Service on Smart Data Streaming node, to start it I go on the Hana cockpit and select the manage streaming node
Click on SWS and start it
Back to my Raspberry, I need to start the following string command to load my test data:
echo "IOTLAB,i,,15:04:54 PM,35,26,VERY GOOD,43" | /sap/install/bin/streamingconvert -d "," -p iotlab:9093/environment_st | /sap/install/bin/streamingupload -p iotlab:9093/environment_st
Thus, we can see my insert successfully proceed and from a process standpoint that my connection has been initiated
Now let’s check on the Hana side if my MEXFL table has received the information, and I can see all my input based on the insert command
The load test successfully done, I will achieve the last part of configuration in my Raspberry which consist of creating a custom python script which will retrieve the sensor data.
Retrieve sensor data and load them into Streaming Lite project
For my IoT project I use a GrovePI platform on top of my raspberry for sensor, thus I don’t have any physical sensor connect to my Raspberry.
GrovePI execute python script to use the sensor and retrieve the data, so how to load those data into Streaming Lite??
Once Streaming Lite is deployed on Raspberry, it basically uses XML-RPC internally to communicate which give several options:
- Create a custom java adapter from my hana studio and load it into my Raspberry
- Communicate directly with the XML-RCP internal server to load my sensor input
- Create a custom python script which encapsulate input data and execution command
I have decided to use the third option, from a sequence stand point the first thing to do is to create a special command line at os level.
In the /usr/bin I create the command called “stream” and include the following line
Now done I test my command with the syntax and check the result in Hana
So, my command is working, I will now create a test script to see if the proceed work to in python. I will voluntary put some data for input
Execute my script and check the data in Hana again
It works, after this validation I can now create the real python script which will be use to capture the sensor input and send then into Hana.
Check live data streamed and received in Hana
Below, here is my final script to be used to ship the sensor data captured from GrovePI into Hana
In my above script the line 25, P1 capture the sensor data and ended by a pipe command ( | ) followed by P2, line 26, which convert and upload the data to my smart data streaming server.
In order to control the cadence of data flowing to Hana, I have decided to only read every 10 second the captured data, line 28 time.sleep(10).
And finally, to be able to compare the data read by the script vs what is incoming in Hana I have declared a print, line 27, so I can see in realtime if the data matches.
Let’s run the script now and compare
As we can see the data are read and stored to Hana in my table “MEXFL” properly, I can follow up with Dynamic Tiering part.
Store the incoming data in Dynamic Tiering
Earlier in my documentation I have installed and configured dynamic tiering, the purpose of this is to avoid having a huge amount of data loaded in memory.
To store my MEXFL table in DT I will basically right click on it and mark as “extended”
Note : I have let me sensor for a moment in order to capture enough data before to store my table in the extended storage.
Moving table in extend storage is a very simple move, however you will need to take some consideration such as the volume of data to move and the disk write speed performance.
For my lab I’m using SAS storage with 10k speed and also I’m controlling the frequency of the data sending (every 10 sec), for a real life case scenario consider to read the following document for guidance and best practice recommendation
I am done with the DT part now, so I will go to the next part of my documentation with the creation of my custom application.
Build a custom application for my project
To expose and visualize my data I will create a basic XS application, SAP HANA XS can be used to expose data from Tables, Views and Modeling Views to UI layer.
There are major 2 ways to expose data the data to clients – OData and XSJS, in my case I will use the OData option.
I will briefly explain my step but won’t go deep since it’s not the main goal of this documentation, how do I proceed? here are the step
I start to build the logic (calculation views), for my setup I have created one calc view per application which will be used for my xsodata services
Once my calc view are all set, I create my xsapp and 3 folders such as:
- “apps” – where I store my xswidget that will contain my dynamic application
- “catalog” – where I store my xsappsite, which contain the catalog for application
- “odata” – where i store my xsodata service used for each individual dynamic application
Once my build completed this is what my application look like
My setup is completed for this part, in my next blog will explain how to configure HSR for this particular setup with Dynamic Tiering and Smart Data Streaming.
- SAP Managed Tags:
5 Comments
