Good things come in small packages: MQTT from Python

How to connect from Python to SAP Cloud Platform Internet of Things (SCP IoT) by MQTT protocol

Today a quick read article as this topic just crossed my way and is kind of unplanned:　As you might know already MQTT is a great protocol to communicate from edge computing devices to gateways submitting small chunks of information from sensors, to actors etc.

You will find good articles already out there on this topic! However, when I started to write the code and wanted to connect to SCP IoT from Python reality was different to what I expected from studying the books! ?

So I want to save those some time that plan to write a Python client with the paho-mqtt package!

Prerequisites

This blog is about the Cloud Foundry IoT (not Neo!) and the help documents I recommend for further insights are here:

SAP Internet of Things Gateway: MQTT

There is as well as a good tutorial for setting up the capabilities, sensors, devices in the cockpit:

SAP Developers Tutorial: Send Data with MQTT

Please read over these first, as I will just cover the (tricky) authentication and Python part.

Authentication

SAP’s IoT framework requires certificate authentication (for those of you interested how to use it in the SAP Netweaver ABAP world you can check my recent blog about it).

For our case we want to download the device certificate generated by the platform in PEM format. You get it from here:

Internet of Things Cockpit: Device certificate generation

You will now get a certificate in .pem-format as well as a pop-up window with a certificate secret which you need to store somewhere (for the moment). If you lose it, you have to create another certificate (which you can do as often as you need it).

Internet of Things Cockpit: Certificate secret

You might ask: What do I need it for, why do I get a “secret”? A PEM file served by the SAP IoT framework contains two pieces of information:

1. The SAP server-side certificate which is public. With this certificate all data which we receive from the IoT framework on our device is decrypted and therefore verified. That is, if we trust the certificate – for that the certificate authority (CA) must be known to both parties. However, this is not scope of our setup.
2. the device’s private key. It is used to encrypt communication to the SAP IoT framework MQTT server. The server holds the public key of the device (as we obtained the .pem-file from the server).

Now here’s the catch: Private keys should be protected against loss. With the key in possession communication can be cloned by any other device and will be authenticated against the server. Therefore, SAP encrypts the private key in the .pem-file and one needs the passphrase (aka: secret) to unlock the key.

Problem for us: The Python MQTT client will ask us anytime for that key it powers up. And it can’t be fed into the program itself (well, it’s a security feature, right?). Fact is, very likely we want to get rid of the passphrase and accept that we have to protect the .pem-file! ?

You need OpenSSL installed for that – it’s three steps. First we extract the private key into a separate file and remove the passphrase protection.

openssl rsa -in <our downloaded>.pem -out private.key -passin pass:<the secret we got>

Then in the second step, we extract the public certificate into a new file.

openssl x509 -inform PEM -in <our downloaded>.pem -outform PEM -out certificate.crt

Lastly, we can combine the two separate files into one.

cat private.key certificate.crt > newcert.pem

Voilà! We got a passphrase-free file for our Python client.

The Python client

The client itself is almost self-explanatory, we need to instantiate it with the alternate ID of the device which is maybe a little special:

client = mqtt.Client(sapIotDeviceID) 

And of course we make use of our new certificate.

import paho.mqtt.client as mqtt
import ssl
import time
import random # to get some 'realistic numbers' :-)

# Parameters
sapIotDeviceID = "<your device alternate ID>"
pemCertFilePath = "./certificates/newcert.pem"
mqttServerUrl = "xyzabc.eu10.cp.iot.sap" # enter the IoT cockpit url here
mqttServerPort = 8883 # Port used by SAP IoT
ackTopicLevel = "ack/" 
measuresTopicLevel = "measures/"

# dummy message
dummyMsg = '{{ "capabilityAlternateId": "<your id>", "sensorAlternateId": "<your sensor id>", "measures": [{{"speed": "{}"}}] }}'

# This function gives a connection response from the server
def onConnect(client, userdata, flags, rc):
    rcList = {
        0: "Connection successful",
        1: "Connection refused - incorrect protocol version",
        2: "Connection refused - invalid client identifier",
        3: "Connection refused - server unavailable",
        4: "Connection refused - bad username or password",
        5: "Connection refused",
    }
    print(rcList.get(rc, "Unknown server connection return code {}.".format(rc)))

# The callback for when a PUBLISH message is received from the server.
def onMessage(client, userdata, msg):
    print(msg.topic + " " + str(msg.payload))

# Send message to SAP MQTT Server
def sendMessage(client, deviceID, messageContentJson):
    time.sleep(random.randint(0,2)) #Wait some random time to make it look realistic
    client.publish(deviceID, messageContentJson)

print("Starting up...")
client = mqtt.Client(sapIotDeviceID) 
client.on_connect = onConnect
client.on_message = onMessage
client.tls_set(certfile=pemCertFilePath, cert_reqs=ssl.CERT_REQUIRED, tls_version=ssl.PROTOCOL_TLS, ciphers=None)
client.connect(mqttServerUrl, mqttServerPort)
client.subscribe(ackTopicLevel+sapIotDeviceID) #Subscribe to device ack topic (feedback given from SAP IoT MQTT Server)
client.loop_start() #Listening loop start 

for _ in range(10): # We send 10 random values
    sendMessage(client, measuresTopicLevel+sapIotDeviceID, dummyMsg.format(random.randint(10,50)))
time.sleep(2) # wait until we have all feedback messages from the server
client.loop_stop

Closing notes

We looked into building an MQTT client with Python that can speak to SAP Cloud Platform Internet of Things. Please note the setup requires ssl package to be imported.

It’s maybe also worth mentioning that the paho-mqtt package is not enabled for multiprocessing. Which means your code must manage the amount of messages you want to send carefully. There are some who launched a new client with every process.

Let me know if you have any suggestions or experiences how to handle multiprocessing (not threading!). Looking forward to your feedback. ?