Produce/consume messages in KAFKA with SAP Netweaver using Java Connector – Part 1/3
Is it possible to produce/send a message from SAP to KAFKA using ABAP? What about is it possible to consume/receive a message in SAP from KAFKA?
In this blog I’ll demonstrate how it is possible to do this using the SAP Java Connector and the Apache Java API.
We live in a world where we need to reduce data friction between systems. Let’s suppose you are asked to send the details (say a business partner) out to a Kafka topic as changes happen inside SAP. As a developer you’ve located the perfect user exit/BADI that gets triggered on/after save where you can put your ABAP code. Now you are stuck. How do you go from taking all the data in ABAP memory, and pass it to KAFKA?
Kafka cannot receive HTTP calls out of the box. You need to use a Java API, or a third party API, or setup an intermediate server to translate HTTP calls to call Kafka.
What are my options?
There are heaps of options to produce/consume messages in KAFKA. Some free, others are not, all with their pros and cons. Apache has a Java client, and Confluent.io has a tonnes of options – .NET, Go, Python etc. There are also heaps of third party options to support other languages as well.
ABAP to Kafka connector? Well, you are out of luck here (at least at the time of this article).
The obvious choice for the unsupported languages is to host a HTTP/REST API for any systems to call. Confluent has a REST support as well.
There are also options where you can stream straight from the database/HANA/hadoop as well. I’m not quite sure how that’s going to work for SAP data, where the data is spread across multiple tables and don’t always make sense to the outside world.
You can spend days exploring the different options, so choose the one that suits.
What about standard SAP and third party enterprise products?
SAP of course have their standard solutions, such as SAP Data Hub, SAP HANA Connector, and possibly others. I have not tried them. There are also other third party companies that provide enterprise solutions at a cost, such as Ad. Google it or search on SDN, all with their pros and cons.
When I first started, I was hoping there could be a standard SAP KAFKA class that I can call to send a message, but there isn’t one. I’ve read a blog that talked about using ABAP Channels but it didn’t work for us.
- I’m not here to say which solution is better as I don’t know. My intention is not to compete with the various products/solutions out there either. As a techie, I just want to see what is possible. To me this option might be the cheapest one without the need of extra hardware or too much setup.
- This is not a tutorial on Java, or KAFKA, AWS, or how to run your own Netweaver environment. It does require knowledge on all of these to follow what I’m doing, so let’s hope you can follow along.
- This is also not a production scale demo. There are many tweaks you can do to make this scale.
- This demo will cost you money for cloud hosting if you were to follow.
Hopefully some of you will find this interesting. It took me a couple of nights to understand how to setup a JCO server properly as the standard SAP documentation is not the best.
What am I going to do?
- Setup a demo Netweaver system using SAP CAL (Cloud Appliance Library) in AWS
- Setup some tools that I’m going to use in the EC2 instance
- Setup KAFKA
- Setup SAP Java Connector (JCo)
- Create/configure SAP Java Connector Server
- Produce a message to a KAFKA topic from ABAP
- Consume a message from a KAFKA topic and call ABAP
- KAFKA – 2.12
- SAP Java Connector (JCo) – 3.0
- SAP NetWeaver AS ABAP 7.51 SP02 on ASE
I’ve put my code on github…. you might find it useful.
Setup a demo NW system in SAP CAL
If you do this, it will cost you money!!!
SAP Cloud Appliance Library (CAL)
Go to SAP Cloud Appliance Library – https://cal.sap.com/, register and create a new “SAP Netweaver AS ABAP 7.51 SP02 on ASE” instance on AWS. You will be presented with many solution choices, so you can pretty much choose any NW instances available. Some options are slightly cheaper than others. You can choose other cloud providers like Google Cloud or Azure. Like I said I’m not here teach you how to use CAL here, so I’m sure you can google it and find out.
When I first tried to do so, it didn’t work, but a day later I got an email and I went in again and it worked again. I can choose from 113 solutions, it’s really quite fascinating!
Download the Key and save it somewhere (Don’t lose it)
Remember the password you entered to create the instance with CAL. There’s no way of recovering (that I know of), and you’ll need to that to use the SAP system.
Amazon AWS EC2
As you can see there are two instances created. These are not covered in Amazon AWS free tier so they will cost you some money.
After a while, I have 2x AWS EC2 instances:
- a SUSE Enterprise Linux server instance installed with the SAP Netweaver with HANA backend, installed and ready to go.
- a Windows instance with SAPGUI and all the development tools you’ll need (you may or may not need this, you can delete it to save some money if you got all the tools installed on your laptop already)
- I have SAPGUI installed on my laptop
- I’ve attached an elastic IP to my instances so that the Public DNS and IP does not change after EC2 restart.
- Turn of your EC2 instances if you are not using it. It cost less than USD$0.50 per hour but it does add up over time.
- Setup cost alarms in AWS to warn you if you go over your budget.
- I’ve setup a AWS CloudWatch event that calls a Lambda function to switch off my EC2 instances every hour to safe cost. Yes it’s inconvenient but I make sure I don’t get a nasty surprise.
- Again, this is not a tutorial about AWS, so go google it and I’m sure you can find out 😀
SSH into the Linux instance
You can use PUTTY (for Windows) or just terminal for Mac. I’m sure you can work out how to do this with a quick googling around.
Now notice the user created is “root”, and password is the same password you entered for CAL when you created the instance. You need to provide the key/cert to log in.
For Putty you will need to convert the cert using PUTTYGEN first.
See you in part 2!
Leave me some comments so I know how I went.