This presentation was first presented at TechEd in 2015 and has now been chosen to be highlighted as an SAP TechEd Lecture of the Week. The focus of the presentation is on end to end architecture and integration of real-time complex event processing and streaming analytics for Internet of Things scenarios. Several HANA platform components are highlighted including HANA smart data streaming, HANA remote data sync, HANA dynamic tiering, HANA Vora and of course our core HANA in-memory database.
Since last October, we have made enhancements in both the SP11 and SP12 releases of HANA smart data streaming that further strengthen the end to end integration.
Streaming Lite Integration with SQL Anywhere
In SP11 we introduced a SQL Anywhere database adapter for Streaming Lite that enables Streaming Lite projects to interact directly with a SQL Anywhere database on the edge. This allows Streaming Lite to leverage remote SQL Anywhere databases that synchronize data bidirectionally with the HANA consolidated database to:
- Access reference data from the SQL Anywhere database to use within the streaming project
- Write out data to the SQL Anywhere database for later synchronization when working in edge processing situations with intermittent or occasional connections back to the core HANA system
Enhancements to the HANA Output Adapter
The HANA Output Adapter for HANA smart data streaming has always been a robust, high performance adapter able to output data to HANA at rates of over 2 million events or records per second. In the SP11 release enhancements were made to simplify the configuration of the HANA Output Adapter and in SP12 further enhancements were made to improve the reconnect capabilities in case of lost connections and an additional performance configuration option to control the rate at which commits are issued against the HANA database.
Enhancements to the IQ Output Adapter
SAP IQ is a key component for handling petabyte volumes of structured data as part of an end to end IoT architecture. Two important enhancements have been made to the IQ Output Adapter for HANA smart data streaming in our SP12 release.
- Support for server-side loads by the IQ LOAD TABLE statement. This enhancement leverages shared storage between HANA smart data streaming and IQ to reduce network overhead when loading output data from HANA smart data streaming into IQ.
- Support for the IGNORE CONSTRAINTS option of the IQ LOAD TABLE statement. This option is particularly useful when a "send at least once" architecture is implemented. In a "send at least once" architecture, guaranteed delivery is implemented by resending records if there is any question that it may not have already reached the destination. Combining a "send at least once" architecture with the IGNORE CONSTRAINTS option allows the system to impose "exactly once" semantics at the point where the data is persisted into IQ.
Introduction of Kafka Adapters
Kafka is a popular open source message streaming system. The introduction of Kafka input and output adapters in SP12 simplifies the integration of existing Kafka systems with the industry leading streaming analytics capabilities of HANA smart data streaming and the rest of the HANA platform.
