Near real-time SAP HANA data replication with SAP Data Intelligence

[architectSAP]

Out of the box SAP Data Intelligence comes with a trigger based near real-time data replication mechanism for SAP HANA similar to the SAP Landscape Transformation Replication Server.

The magic lies within the SAP HANA Monitor operator. It watches an SAP HANA table and outputs newly inserted rows as a message by creating a trigger on the table to copy each inserted row into a temporary table. The temporary table is periodically queried and cleared so that it contains the new rows since the last poll. No output is produced if there were no new rows since the last query:

As an example I use my IoT scenario from Restore your NEO Internet of Things Service scenarios with Apache Kafka and the SAP Data Hub, now of course leveraging my SAP Cloud Platform Trial Account on Cloud Foundry:

When I look at the table definition, I see that a trigger statement has been added:

CREATE COLUMN TABLE "DBADMIN"."T_IOT_985EAC8DCBD198C302F5"(

	"G_DEVICE" NVARCHAR(255),

	"G_CREATED" LONGDATE,

	"C_TEMPERATURE" DOUBLE

)

UNLOAD PRIORITY 5 AUTO MERGE;

CREATE TRIGGER "DBADMIN"."T_IOT_985EAC8DCBD198C302F5_314021906" AFTER INSERT ON "DBADMIN"."T_IOT_985EAC8DCBD198C302F5" REFERENCING NEW ROW NR FOR EACH ROW 

BEGIN INSERT INTO "T_IOT_985EAC8DCBD198C302F5_TEMP_284535295"(G_CREATED,C_TEMPERATURE) VALUES (:nr.G_CREATED,:nr.C_TEMPERATURE); END;

And subsequently, with each reading added into my IoT table, I get a respective message:

So, this is in fact no magic but works incredibly well without any manual trigger creation or temporary table management.