Putting the Kafka broker to work in SAP HANA Smart Data Streaming
In a previous blog, our very own Jeff Wootton compared SAP HANA smart data streaming to the Apache Kafka message broker. The key takeaway of that blog post is that while there are certain similarities between the Kafka broker and HANA SDS, there is also a key difference that emphasizes the success of using these two technologies in conjunction with one another.
With just the Kafka broker, you move messages (unchanged) from a producer to a consumer. But bring smart data streaming into the picture and now you can take that raw data, apply business logic to it, transform it into meaningful and actionable information, and then send it to the consumer.
With the release of SP 12, HANA SDS now includes 10 adapters that work with the Kafka broker so you can put these two technologies to work for you. They are:
|
Adapter |
What it does |
|
Kafka Avro Record Input adapter |
Obtains data in Avro record binary format from a Kafka server and publishes it to HANA SDS. |
|
Kafka Avro Record Output adapter |
Reads data from HANA SDS, formats it to Avro format, and writes it to a Kafka server. |
|
Kafka CSV Input adapter |
Obtains CSV data from the Kafka server and publishes it to HANA SDS. |
|
Kafka CSV Output adapter |
Reads data from HANA SDS, transforms it to CSV format, and writes it to a Kafka server. |
|
Kafka Event XML Input adapter |
Obtains event XML data from a Kafka server and publishes it to HANA SDS. |
|
Kafka Event XML Output adapter |
Reads data from HANA SDS, formats it to event XML format, and writes it to a Kafka server. |
|
Kafka JSON Input adapter |
Obtains JSON data from a Kafka server and publishes it to HANA SDS. |
|
Kafka JSON Output adapter |
Reads data from HANA SDS, formats it to JSON format, and writes it to a Kafka server. |
|
Kafka String Input adapter |
Obtains string data from a Kafka server and publishes it to HANA SDS. |
|
Kafka String Output adapter |
Reads data from HANA SDS, formats it to string format, and writes it to a Kafka server. |
If you want to build a custom Kafka adapter using the adapter toolkit, you can use these new transporters and formatters:
|
Name |
What it does |
|
|
Kafka Input transporter |
|
|
|
Kafka Output transporter |
|
|
|
Avro Record to ESP formatter |
|
|
|
ESP to Avro Record formatter |
|
|
|
String to ESP column formatter |
Passes a single input string to a single smart data streaming column. |
|
|
ESP Column to String formatter |
Passes a single smart data streaming column to a single input string. |
The String to ESP column and ESP Column to String formatters are simple formatters with no additional properties aside from common ones to connect them from one module to the next. There are no additional details on these formatters in the documentation.
For detailed information on the rest of the new transporters and formatters, see the Transporters Currently Available from SAP and Formatters Currently Available from SAP sections in the SAP HANA Smart Data Streaming: Building Custom Adapters guide.
For detailed information on configuring and using these adapters, see the Kafka Adapter section in the SAP HANA Smart Data Streaming: Adapters Guide.