This blog will help you to set up kafka server on cloud and test it with sap cpi. Recently SAP has released a new adapter called Kafka for sap cpi to integrate with Kafka. Please note that this blog is only for learning purpose, as I have not used this in productive systems.
1. Create Kafka Instance
I did a quick google search to check if there is any kafka trial server. I was lucky enough to find one which was pretty easy to set up and use.You can create your own kafka instance by subscribing.
Once you are done with the service setup you will get a kafka cluster instance. Please make note of kafka connection information.
You have to create your topic. For example, I have created a topic called hello_world.
2. Test Producer and consumer using Java
First I have tried to check If I can produce and consume kafka messages using java as it was straightforward which will confirm that everyhing works with kafka server setup.
Follow this link for producing and consuming message using java.
1.First download key and certificates and Create keystore and trust store using openssl.
2.Generate the private key and trust store.
Now create a java project and run first producer class after that run consumer class. Once you have run the consumer class, it will keep polling the topic.
You can check the number of produced messages and consumer groups.
3. Connect to Kafka server using SAP CPI
1. Connection Test
You can do the connection test first before deploying your flow, to make sure that connectivity is functioning. I have uploaded the key pair(.p12)file into the keystore.
Now you can do the connection test and you should be able to connect successfully.
Download the certificates and upload it into key store.
With this we are done with connection set up.
2. Receiver scenario (SFTP to kafka)
I have created a very basic SFTP to Kafka iflow for sending messages to kafka topic. You can also make it more simple by using time with content modifier to send some data to kafka instread of sftp sender.
Host and private key alias, topic name.
3. Sender Scenario (Kafka to SFTP)
File in sender folder
Records are read from kafka server and stored in sftp folder.
kept sftp sender to poll the files once a minute and send the data to kafka topic. kafka test receiver flow will send the data to kafka. kafka test sender flow will read the data from akfa and create file in sftp. messages are processed successfully.
consumer group with sender iflow id is created on kafka side and polls the records.
Hope this can help you set up kafka server and test with cpi.