I find that one of the great things about working at SAP is the vast amount of tools that we have at our disposal. When looking to create new solutions and to explore what we can do, it helps to have some cool tools to use. And we definitely have some cool ones in our toolbox.
With the internet of things being the hot topic these days, a lot of discussions tend to gravitate towards machine and sensor data. It makes sense as it's a pretty obvious use case for this. So I wondered what it would be like to create my own sensor and run it through the SAP Real Time Data Platform. Of course that made me ask myself the age old question:
"How hard can it be?"
Turns out that after brushing up on my basic Python and C++, it's not all that bad. The setup for my first example is fairly simple. I have a sensor that feeds SAP ESP, and in turn feeds SAP HANA. I also had SAP Design Studio subscribe to ESP in order to update some charts in real-time. So let's break it down a bit.
Step 1: Sense Something.
Naturally the first step when trying to analyze sensor data is to get a sensor! There's a lot of different ones to choose from, but for this project I decided to go with the Arduino platform. I was a fan of this because it was fairly easy to pick up, the code tends to be very light weight, and it's incredibly modular so it opens up a lot of possibilities for all different types of sensor configurations. And in case you're wondering what an Arduino looks like, it's the green device on the left in the picture below.
For the eagle eyed (and Arduino/sensor familiar), the sensor that I have built here has two probes. The little black one on the lower right corner of the breadboard is a temperature sensor. The square external probe that is wired up to the smaller sensor actually measures moisture levels. The Arduino sketch that I had written and loaded up, at its core, just takes a measurement of the temperature and moisture sensors every second and outputs it to the serial bus. Pretty simple stuff!
This information is then sent over for processing by both a Python script, and SAP ESP. Which brings us to the next component.
Step 2: Stream Something.
For this example, we needed something to take the data output from the Arduino sketch, and add some extra metadata (timestamp, sensor ID number, etc.), and send it over to ESP. A Python go-between wasn't necessarily required, but it does add a bit of flexibility for us to use Python's processing capabilities to enrich the data before sending it over to ESP. And while Arduino can do the processing as well, I figured that having an external program to do more of the processing would be a general better practice as the Arduino's memory space is limited and it's probably best to keep its code as efficient as possible. The Python code that I used sends the data to a socket server that ESP is running that is listening for CSV inputs.
The ESP stream has a socket server input, which takes the CSV data from the Python program, and does some basic type conversions, and streams it to a SAP HANA table. The CCL notation diagram is below. Again, simple stuff.
The HANA output allows us to constantly update a table for bulk analysis. For real time stream analysis, loading it into HANA isn't required, but it does allow us to perform historical analysis on the life of the sensor.
So we got a sensor producting data. We have a means to fetch and process this data in real time. So, what do we do with it?
Step 3: Visualize Something.
As of SAP Design Studio 1.4, there were new components introduced for real time data analysis. In essence what this does is creates a DS adapter which hooks into the web services server on ESP. DS then pings the WS server at a pre-defined time interval to poll for new data in the ESP stream, and then updates whichever DS charts are using it as a data source. This version also introduced a new chart which shows a time-based line chart. The difference between this and the standard line chart is detailed in an SCN post (http://scn.sap.com/community/businessobjects-design-studio/blog/2015/03/29/building-real-time-visual...). At the end of the day though, the real-time streaming data source allows us to update all the different DS chart types as the information streams in through ESP, which opens up possibilities for the real-time analysis of all types of streaming data.
So, while all this is great and wonderful, what does it look like when put together? Below is a short video that I've put together that runs through the execution process of everything that I've talked about. It will show the sensor itself, the live connection to ESP and HANA, and a quick example of real time charts in Design Studio.
So that's a brief overview into my first foray into building my own sensor and linking it up to the SAP data platform. Over time I'm hoping to increase the level of sophistication of the Arduino sensors, tie them closer to real-life use cases, and fine tune the dashboards for real-time analysis as well. It's also been a really interesting experience building out a sensor (and attempting to install and configure ESP on my own), and hopefully this experience can help someone out there in the sensor/streaming space as well!
- SAP Managed Tags:
