Skip to Content
Technical Articles
Author's profile photo Jacky Liu

Develope kafka rest api with nodejs to make it callable from CPI through sap cloud connector Part I

Sap cloud integration has kafka adapter , but currently the kafka adapter does not support  calling kafka on premise  through SAP Cloud connector . We can refer to note 316484 for this . There are 2 options for resolving this currently :

1,  Expost the kafka on premise to public web, then CPI kafka adapter can connect to it .

2,  Build rest api proxy for kafka , then CPI can produce or consume message on kafka op by using CPI http adapter with the help of sap cloud connector .

Today I want try to investigate option 2 . There are existing open source kafka rest proxies, but maybe customer has some concern to use them . In this blog I will try to  develope kafka rest api with nodejs to make it callable from CPI through sap cloud connector . In this part I , I will  prepare kafka  envirement .

To build the scenario, first  let me use docker to create kafka service  on my  computer . Of course I have installed docker on my laptop . The following is the steps .

Step 1 : pull docker image for zookeeper and kafka

docker pull wurstmeister/zookeeper

docker pull wurstmeister/kafka

Step 2 : Create and start zookeeper container

docker run -d –name zookeeper -p 2181:2181 wurstmeister/zookeeper

Step 3 : Create and start kafka container and connect kafka to zookeeper container in step 2

docker run -d –name kafka –publish 9092:9092 –link zookeeper –env KAFKA_ZOOKEEPER_CONNECT=zookeeper:2181 –env KAFKA_ADVERTISED_HOST_NAME=localhost –env KAFKA_ADVERTISED_PORT=9092 wurstmeister/kafka

Step 4 : Enter kafka container

docker ps

docker exec -it 664dcfafd35c /bin/bash


Step 5  : create and check topic in  kafka container –create –zookeeper zookeeper:2181 –replication-factor 1 –partitions 1 –topic dblab01 –list –zookeeper zookeeper:2181

Step 6  : start producer and produce data in  kafka container –broker-list localhost:9092 –topic dblab01

Step 7  : start consumer and consume data in  kafka container

Step 8 :  start server.js to to test  kafka locally  .

The following is the code .

const { Kafka } = require('kafkajs')
const kafka = new Kafka({
  clientId: 'my-app',
  requestTimeout: 25000,
  connectionTimeout: 30000,
  retry: {
    initialRetryTime: 3000,
    retries: 0
  brokers: ['localhost:9092']
const producer = kafka.producer()
const consumer = kafka.consumer({ groupId: 'test-group' })
const run = async () => {
  // Consuming
  await consumer.connect()
  await consumer.subscribe({ topic: 'dblab01', fromBeginning: true })
    eachMessage: async ({ topic, partition, message }) => {
        // await new Promise(r=>setTimeout(r,3000))
        offset: message.offset,
        value: message.value.toString(),

setInterval(async ()=>{
    await producer.connect()
    await producer.send({
        topic: 'dblab01',
        messages: [
          { value: 'Hello KafkaJS user!' },

npm install kafkajs

node server.js


to be continue in Part II .


Best regards!


Jacky Liu




Assigned Tags

      Be the first to leave a comment
      You must be Logged on to comment or reply to a post.