Produce/consume messages in KAFKA with SAP Netweaver using Java Connector – Part 2/3

This is the second part to demonstrating how to get SAP Netweaver to produce/consume messages in KAFKA from SAP.

Link to part 1 Part 1

 

If you are just keen to see the results, go to Part 3.

 

Below will cover the following:

• Connect to your SAP server using SAPGUI
  • Install license
  • Developer Key
• Setting up tools
  • Java Development Kit
  • Nano
• Setting up KAFKA
• Setting up SAP Java Connector (JCo)

 

Just to reiterate, I didn’t use the windows EC2 instance that had all the developer tools. I only used the SUSE linux instance installed with the SAP Netweaver backend.

 

 

Source code

I’ve put my code on github…. you might find it useful.

 

 

Setup your SAP system

You can either use your windows EC2 instance’s SAPGUI and it is already setup with a system to connect to.

If you have your own SAPGUI installed on your own machine. The application server should point to your elastic IP of your EC2 instance, or else it may change after every EC2 restart.

 

First log in

Client 001

User: DEVELOPER

Password: What you entered when you created your CAL instance

 

Install trial license

See here , or google how to get a minisap license.

In a nutshell, get your hardware key from transaction transaction SLICENSE.

Go to https://go.support.sap.com/minisap/#/minisap and generate a license for A4H, enter your hardware key.

Upload it to SLICENSE and you are done.

 

Developer key

The user DEVELOPER should already have a developer key assigned, so in case you are wondering, you don’t need one. Just don’t go and create your own user and expect to be able to get a developer key for it (I found out the hard way).

 

 

 

 

Setting up tools

First you need to make sure you can SSH into your EC2 instance.

JDK

Both KAFKA and SAP Java Connector requires Java, so it makes sense to start installing that first.

From memory a Java 1.6 comes with the EC2 instance but it is too old for KAFKA, so we need a newer one.

To install

Run command:

sudo zypper --non-interactive install java-1_8_0-openjdk-devel

To test:

Run Command:

java -version

Run Command:

javac

If they didn’t return any error, you are good to go.

 

Nano

This step is entirely optional. I need an editor in linux, and I like to use nano. You can use whatever you like such as vi.

To install

Run Command:

wget https://download.opensuse.org/repositories/openSUSE:/Leap:/15.1/standard/x86_64/nano-2.9.6-lp151.2.3.x86_64.rpm

sudo rpm -i nano-2.9.6-lp151.2.3.x86_64.rpm

If this link does not work (because I have no control over it, search for somewhere that that might have a nano rpm distribution)

 

To test:

nano <any file>

 

To exit:

Ctrl + X

 

 

Setting up KAFKA

Install

Run Command:

curl -O http://apache.mirror.amaze.com.au/kafka/2.4.1/kafka_2.12-2.4.1.tgz

tar -xvf kafka_2.12-2.4.1.tgz

rm -rf kafka_2.12-2.4.1.tgz

sudo ln -s ~/kafka_2.12-2.4.1 /opt/kafka

 

Let’s explain

The script will first go to Apache and download KAFKA. If the link in the above command does not work, then change it to the latest from Apache.

The EC2 instance will download KAFKA tgz file, untar it and we create a symbolic link /opt/kafka to the decompressed kafka directory.

That’s it… now KAKFA is installed!

 

 

Configure

We are going with the most basic configuration/setup here, with single zookeeper, single KAFKA server. It’s up to you to explore more complex setup.

Now I’m going to refer to the Apache quick start guide since I can’t explain it any better and there’s no point of me repeating either.

In short, we are going setup one zoo keeper (on port 2181), one KAFKA server (on port 9092).  Remember these ports as you will see code referring to them later.

I run the following script to start the zookeeper and KAFKA server. You need to repeat this after each restart.

#!/bin/bash
export KAFKA_HOME="/opt/kafka"
export KAFKA_HEAP_OPTS="-Xmx512M -Xms256M"
nohup sudo $KAFKA_HOME/bin/zookeeper-server-start.sh $KAFKA_HOME/config/zookeeper.properties > /dev/null 2>&1 &
sleep 2
nohup  $KAFKA_HOME/bin/kafka-server-start.sh  $KAFKA_HOME/config/server.properties > /dev/null 2>&1 &
sleep 2

Note that I set the KAFKA heap to be smaller as the EC2 instance struggled with the default memory settings.

Remember to chmod it to execute.

chmod +x kafka_start.sh

See code here

Create a topic:

I’m going to create a topic “my-kafka-topic”. You can call it whatever you want. You only need to do this once. If you get an error it means your zookeeper or KAFKA server is not started properly.

/opt/kafka/bin/kafka-topics.sh --create --topic my-kafka-topic --zookeeper localhost:2181 --partitions 1 --replication-factor 1

 

Test it

Produce a message

/opt/kafka/bin/kafka-console-producer.sh --broker-list localhost:9092 --topic my-kafka-topic

 

Consume the message

In another SSH session, run this:

/opt/kafka/bin/kafka-console-consumer.sh --bootstrap-server localhost:9092 --topic my-kafka-topic --from-beginning

 

 

Setting up SAP JCo

Make sure you get JCo 3.0 and not 3.1

I paid a hefty price of trying to get version 3.1 to work, only to wasted hours on trying to fix a ARPCRFC fast serialization error. So I’m going to use version 3.0 here

 

Where to download

https://support.sap.com/en/product/connectors/jco.html

 

Install

Transfer and unzip the file to the EC2 instance

I suggest you download the “sapjco30P_20-10005328.zip” file first in your desktop and upload it to the EC2 instance. I suggest you either put it somewhere public on the internet and use “wget” or “curl” in the SSH terminal to download. I put it in an S3 bucket and use AWS CLI commands to retrieve it – do whatever works for you.

After the file is on the server, you should unzip it to the folder sapjco30.

 

mkdir sapjco30
cd sapjco30
aws s3 cp s3://<my-s3-bucket>/sapjco30P_20-10005328.zip sapjco30P_20-10005328.zip
unzip sapjco30P_20-10005328.zip
tar -xvf sapjco30P_20-10005328.tgz

If you are not using S3 bucket for transfer, skip the s3 line.

That’s it. SAP Java Connector is ready to be used!

Your directory should look like this.

 

Now let’s get the simple JCo Server to work. JCo server is basically the receiver of an RFC calls from any NW system. You need to first register this “JCo Server” with the SAP system, kind of like a handshake, so that the NW system knows where to call.

 

Create RFC Destination

Name	ZMP_JCO_SERVER
Type	T
Program ID	ZMP_JCO_SERVER
Activation type	Registered Server Program

 

Test the RFC Destination

It should return an error for now… don’t worry… it will work later on.

The error means it can’t connect to the JCo server.

 

Configure the JCo Server

I’m going to use the stock standard JCo Server example provided by SAP. You can make this a lot more complex and scalable and performant.

Create the JCo Destination file

This bit took me hours to figure out, as I couldn’t find any documentation that explains this. Perhaps it is straight forward. Standard HTML documentation that comes with the zip file shows you the code and assume you to know how to run it.

This tells the JCo Server code where our SAP system is. The JCo Server will use this to call SAP system to register a program ID.

• EXT_SERVER.jcoServer

#A sample confidguration for a standard registered server 
#Replace the dummy values and put this file in the working directory.
#In eclipse environment it is the project root directory.
jco.server.connection_count=2
jco.server.gwhost=localhost
jco.server.progid=ZMP_JCO_SERVER
jco.server.gwserv=sapgw00
jco.server.repository_destination=ABAP_AS1

Make sure you enter the same program Id as the RFC destination file.

For the registration, the ABAP_AS1 is required as the JCo server makes a call to SAP.

 

• ABAP_AS1.jcoDestination

#Sample destination configuration when using an application server over CPIC
#Replace the dummy values and put this file in the working directory.
#In eclipse environment it is the project root directory.
jco.client.ashost=vhcalnplci
jco.client.sysnr=00
jco.client.client=<sap client>
jco.client.user=<sap user>
jco.client.passwd=<sap password>
jco.client.lang=en 

Change your ashost to the server name. localhost might work, haven’t tried.

 

Compile the jco server example:

Edit the StepByStepServer.java file that comes with the Jco zip file.

Configure the server name to be the name of your file.

DESTINATION_NAME=Program ID of RFC Destination.

    static String SERVER_NAME1 = "EXT_SERVER";
    static String DESTINATION_NAME1 = "ZMP_JCO_SERVER";

Comment out these lines.

    public static void main(String[] a)
    {
        step1SimpleServer();
//      step2SimpleServer();
//      step3SimpleTRfcServer();
//      step4StaticRepository();
    }

 

Run

javac -cp ~/sapjco30/sapjco3.jar:/opt/kafka/libs/* StepByStepServer.java

java -cp ~/sapjco30/sapjco3.jar:/opt/kafka/libs/*:. StepByStepServer

 

 

 

Test it

Test the RFC destination

It’s working!

Test the function module

Go to transaction SE37 and call function STFC_CONNECTION with the RFC destination.

The call is successful. The Java code response with “Hello World”.

The call is successfully received by the JCo Server.

 

Finally, all the bits and bobs are set up… Now… back to why we are trying to do this, we want to be able for ABAP to call a java program using Java Connector, and produce a message on KAFKA… See Part 3.