Azure Event Hubs
Azure Event Hubs is a fully managed serverless event streaming platform on Azure Cloud that can ingest & distribute huge amounts of events.
Event Hubs is used in many digital initiatives on Azure around the globe due to its elastic and georedundant capabilities, ranging from messaging/eventing to stream data analysis use cases.
Typical challenges of streaming event logs are
- You need to manage checkpoints inside the log of records you already processed.
- You need to manage consumers to distribute processing over 1…n shards (or partitions)
Although Azure Event Hubs offers open protocols like AMQP to a wider audience it doesn’t solute this topics for you.
Aside that also event streaming system can process huge spikes, whereas traditional enterprise systems have limits and need to be shielded from huge inbound processing peaks.
This article shows how to connect Azure Event hubs with SAP PO & KaTe’s Kafka adapter and how checkpoints, high availability & data transformation is managed while sending/receiving event in a SAP landscape.
Setup Azure Eventhubs
A Event Hub on Azure can simply be created via CLI or via azure portal.
Please note that creating a Event Hub might create costs in your Azure billing 😉
Create a Event Hub
E.g. below an Event Hub named “MySampleEventHub” is created in azure portal. Its worldwide DNS domain name by that is: MySampleEventHub.servicebus.windows.net.
Obtain Security credentials and Host
After creating the event hub there is a default entry for shared access policies with the name “RootManageSharedAccessKey” as default access user (you might choose different ones in a realworld scenario)
The entry contains the actual Connection primary key (which is a combination of endpoint and authentication information) needed for authenticating with Azure Event Hubs.
Obtain Security credentials and Host
SSL certificates can simply be obtained via e.g. Portecle from the dns hostname to be imported into SAP NetWeavers TrustedCA keystore. The adapter just uses the TrustedCA keystore for certificates, no external key stores are necessary.
Publishing Events to the Azure Event Hub
The KaTe Kafka adapter can publish events into the Event Hub via a SAP PO receiver channel.
In order to configure a channel, just the hostname and credentials from Azure Event Hubs need to be pasted into the channel configuration.
Azure Event hubs uses a fixed username ($ConnectionString) and the endpoint with its shared secret key as password to authentication with SASL over Kafka APIs.
- Bootstrap Servers: <your event hub dns name> + “:” + port 9093
- Username: $ConnectionString
- Password: Endpoint=<your endpoint and access key copied from azure portal>
The topic in the Event Hub can be configured in the receiver channel (or if needed overridden with ASMA properties at runtime). The event key of a published message is set by default to the SAP PO message id, the partition is chosen automatically. Both also can be overridden via dynamic ASMA properties at runtime.
Optionally the payload format can be transformed from XML to JSON or Avro (with or without a central Avro registry).
Processing Event Hub records
In order to process events from Azure Event Hubs, the Kate Kafka Adapter can be configured as sender channel.
For configuring a channel the connection information (host/username/pwd) can be pasted into the channel configuration.
Optionally different detail settings can be applied like:
- group name and parallelization of consumers of the channel
- timeouts and number of polled records per poll
- batch options to fetch several records in one SAP PO message or simply 1 event record = 1 PO message
- seek options to seek at start/end or a certain position/partition in a stream topic
- data transformation from JSON/Avro to XML (optional)
The beauty of using a Kafka client (as in this adapter) comes with Kafka’s builtin client management on the server side. The adapter spins up one or more consumers per PO server node. The Kafka framework itself “load balances” partitions over those consumers to receive Kafka records. If one consumer or even SAP PO server node goes up or down a rebalance happens to keep processing running.
The checkpoint where in the log each client is positioned at the current moment is saved by the framework itself and doesn’t need any solution on the PO scenario implementation side.
Upon runtime the adapter will poll a fetch of records and (depending) on the settings process them as one SAP PO message per records and then commit the new offset to Azure Event hubs.
The adapter implements Deduplication via record key, so if a fetch is reprocessed the same records will only show up once as SAP PO message.
In Order Processing (EOIO)
In Order Processing of a Event Hub is very simple to solute by this approach. As each partition is glued to exactly one Kafka consumer, each one processes the partition records in order and delivers SAP PO messages in order within its partition which plays hand in hand with SAP PO’s internal EOIO quality of service processing.
Avro Schemas & schema registries
A sender channel also is configurable to transform from JSON to XML & Avro to XML while the records are passed as message into SAP PO.
If you host an own Kafka schema registry sideboard to Azure Event Hubs it could also be used to register known Avro schemas by sender and receiver channels. The schema can be pasted into the channel configuration and will be registered on the topic used in a schema registry also configured in the channel settings.
Azure Event hubs is a widely used event stream solution for digital initiatives on Azure. It’s Kafka compliant client API opens the platform to wider non proprietary usage.
The KaTe Kafka adapter allows you to connect Azure Event Hubs as well as any other Kafka broker from a SAP Landscape via SAP PO with simple configuration steps.
The adapter implements all heavy lifting and all best practices for transactional exactly once processing, deduplication, security and data transformation with Kafka and Azure Event Hubs.
No external keystores for SSL certificates / schema storage for Avro is needed and everything is concealed into SAP NetWeaver standard configuration and security options.
Developers can focus on their integration scenario instead of developing custom solutions to solute Java client libraries, SASL authentication, deduplication or message transformation between XML & JSON or Avro.
Also the adapter allows you to control event spikes for inbound processing into a SAP landscape via common SAP PO configuration & batching options.
As a product the adapter is fully maintained by KaTe GmbH, a SAP partner company with deep SAP expertise and know how in open standards and event processing solutions like Kafka.
The product is kept in sync with all upgrades on Kafka features & SAP PO releases. New features & best practices are introduced continously on feedback by customers.
Azure Event Hubs:
KaTe Kafka Adapter for SAP PO
KaTe Kafka Adapter in SAP Appcenter: