A “Dream Team” on the Edge for the Internet of Things

[former_member191236]

You do not have to look very far to see industry predictions of the huge number of devices that will be connected in the next few years, and the enormous amount of data they will be produced. With data volumes in the petabytes being predicted, there has been a lot of focus on the big data problem, and how technologies like HANA can be used to centrally mine business value out of these new sources of data.

However, when talking of data sizes in the petabytes, it is not easy to dismiss the question of how the data will move from the sensors into the cloud. There will be some environments that simply do not have the fat, fast network pipes required to send high volumes of data in real-time (think off-shore oil rigs and mines). In these environments the data will need to be stored, analyzed, and acted upon out at the edge near the sensors. Even for environments where all of the data will eventually be stored in the cloud, the real-time requirements of an application may not allow for the additional latency that a round-trip to the cloud will need (think slow and unreliable satellite connections).

For IoT applications that have these requirements, SAP has data management products that have been specifically designed for running at the edge. These do not replace the main IoT processing that is done in HANA, but rather augments it with additional capabilities. Two of these technologies are SAP SQL Anywhere, and SAP Streaming Lite.

SAP SQL Anywhere is a database that has been specifically designed to run in locations outside of the traditional data center, where there is no database administrator. This has made it very suitable for embedding in applications that run in edge location such as water towers and kiosks. SQL Anywhere can embed so well that it is sometimes called an “invisible database”.  With millions of active deployments, it is likely you have used some equipment or system that embedded SQL Anywhere without even knowing it. In addition to the basic functions of storing IoT data, SQL Anywhere provides a rich set of tools for doing data analysis on the edge. Despite the small footprint of less than 20MB, SQL Anywhere provides enterprise features like native spatial queries, full-text search, and OLAP functions.

In some environments, the data volumes and real-time requirements are so high that action must be taken immediately, even before the data reaches the database. For these situations, a solution designed specifically for analyzing high volume and high velocity streams of data is ideal. SAP Streaming Lite is a stream processor for the edge. Like SQL Anywhere, it is designed to run on the types of resource-constrained devices that will deployed in these environments. Streaming Lite is able to take in multiple streams of sensor data and perform queries over them including filtering, calculating moving averages, and detecting anomalies that require attention.

Although SAP SQL Anywhere and SAP Streaming Lite can be used independently, they become a true powerhouse at the edge when they are combined. SAP Streaming Lite becomes the front line for the incoming sensory data. In the case of large data volumes, SAP Streaming Lite can either filter, or average the data to reduce the total amount of data that must be passed on for further processing. In the case of real-time requirements, SAP Streaming Lite can scan the incoming data looking for specific conditions, and raise events to the edge application. Once the data has passed through Streaming Lite, the augmented data can be durably stored in SAP SQL Anywhere for further analysis and querying by the edge application.  The data flow can go the other way as well; historic data from SAP SQL Anywhere can feed back into SAP Streaming Lite to provide historic context and constants.

Although the data may be analyzed at the edge, it may still need to be propagated back to the cloud for the further analysis and auditing. Both edge components have complementary technologies in the data center to facilitate moving the data in and out of HANA. For real-time data, Streaming Lite can connect to HANA Smart Data Streaming to pass the augmented data on for further stream processing in the data center. For historic data, SQL Anywhere can connect to HANA Remote Data Sync to synchronize with HANA whenever a connection is available.

To get started with SQL Anywhere, please download the free SQL Anywhere Developer Edition. The developer edition is fully-featured, not time limited, and free for development use on your desktop. 

For more information on Streaming Lite, which is available as a remote deployment option for SAP HANA smart data streaming, see the Streaming Developer Center.  Users that want to get started can sign up for the SAP HANA Developer Edition, which now includes smart data streaming, and take advantage of this set of tutorials for putting streaming lite into action.

-Written by Eric Farrar-