This presentation was first presented at TechEd in 2015 and has now been chosen to be highlighted as an SAP TechEd Lecture of the Week. The focus of the presentation is on end to end architecture and integration of real-time complex event processing and streaming analytics for Internet of Things scenarios. Several HANA platform components are highlighted including HANA smart data streaming, HANA remote data sync, HANA dynamic tiering, HANA Vora and of course our core HANA in-memory database.


Since last October, we have made enhancements in both the SP11 and SP12 releases of HANA smart data streaming that further strengthen the end to end integration.

Streaming Lite Integration with SQL Anywhere

In SP11 we introduced a SQL Anywhere database adapter for Streaming Lite that enables Streaming Lite projects to interact directly with a SQL Anywhere database on the edge. This allows Streaming Lite to leverage remote SQL Anywhere databases that synchronize data bidirectionally with the HANA consolidated database to:

  • Access reference data from the SQL Anywhere database to use within the streaming project
  • Write out data to the SQL Anywhere database for later synchronization when working in edge processing situations with intermittent or occasional connections back to the core HANA system

Enhancements to the HANA Output Adapter

The HANA Output Adapter for HANA smart data streaming has always been a robust, high performance adapter able to output data to HANA at rates of over 2 million events or records per second. In the SP11 release enhancements were made to simplify the configuration of the HANA Output Adapter and in SP12 further enhancements were made to improve the reconnect capabilities in case of lost connections and an additional performance configuration option to control the rate at which commits are issued against the HANA database.

Enhancements to the IQ Output Adapter

SAP IQ is a key component for handling petabyte volumes of structured data as part of an end to end IoT architecture. Two important enhancements have been made to the IQ Output Adapter for HANA smart data streaming in our SP12 release.

  1. Support for server-side loads by the IQ LOAD TABLE statement. This enhancement leverages shared storage between HANA smart data streaming and IQ to reduce network overhead when loading output data from HANA smart data streaming into IQ.
  2. Support for the IGNORE CONSTRAINTS option of the IQ LOAD TABLE statement. This option is particularly useful when a “send at least once” architecture is implemented. In a “send at least once” architecture, guaranteed delivery is implemented by resending records if there is any question that it may not have already reached the destination. Combining a “send at least once” architecture with the IGNORE CONSTRAINTS option allows the system to impose “exactly once” semantics at the point where the data is persisted into IQ.

Introduction of Kafka Adapters

Kafka is a popular open source message streaming system. The introduction of Kafka input and output adapters in SP12 simplifies the integration of existing Kafka systems with the industry leading streaming analytics capabilities of HANA smart data streaming and the rest of the HANA platform.

To report this post you need to login first.

2 Comments

You must be Logged on to comment or reply to a post.

  1. Vicky Dev

    Hi Robert,

    Can you guide me to any end to end demo on this tool, which seems to be very interesting/exciting. Some documentation or a video showing the tool usage with an end to end scenario would help the global audience.

    Dev.

    (0) 
    1. Robert Waywell Post author

      Hi Vicky,

      Sorry for the slow response.

      For Smart Data Streaming, we have a pretty good set of tutorials available here on the SCN. An overall list is available in the Table of Contents – Smart Data Streaming. This post gives you a learning map starting from ramping up on SDS through to integration with Streaming Lite: Learning how to use Streaming Lite with HANA Smart Data Streaming

      If you are looking for getting started tutorials for Dynamic Tiering, then the SAP HANA Dynamic Tiering Quick Start Guide – Overview is a good starting point.

      Thanks

      Rob

      (0) 

Leave a Reply