Skip to Content
Technical Articles
Author's profile photo Gunter Albrecht

Good things come in small packages: MQTT from Python

How to connect from Python to SAP Cloud Platform Internet of Things (SCP IoT) by MQTT protocol

Today a quick read article as this topic just crossed my way and is kind of unplanned: As you might know already MQTT is a great protocol to communicate from edge computing devices to gateways submitting small chunks of information from sensors, to actors etc.

You will find good articles already out there on this topic! However, when I started to write the code and wanted to connect to SCP IoT from Python reality was different to what I expected from studying the books! ?

So I want to save those some time that plan to write a Python client with the paho-mqtt package!

Prerequisites

This blog is about the Cloud Foundry IoT (not Neo!) and the help documents I recommend for further insights are here:

SAP Internet of Things Gateway: MQTT

There is as well as a good tutorial for setting up the capabilities, sensors, devices in the cockpit:

SAP Developers Tutorial: Send Data with MQTT

Please read over these first, as I will just cover the (tricky) authentication and Python part.

Authentication

SAP’s IoT framework requires certificate authentication (for those of you interested how to use it in the SAP Netweaver ABAP world you can check my recent blog about it).

For our case we want to download the device certificate generated by the platform in PEM format. You get it from here:

Internet%20of%20Things%20Cockpit%3A%20Device%20certificate%20generation

Internet of Things Cockpit: Device certificate generation

You will now get a certificate in .pem-format as well as a pop-up window with a certificate secret which you need to store somewhere (for the moment). If you lose it, you have to create another certificate (which you can do as often as you need it).

Internet%20of%20Things%20Cockpit%3A%20Certificate%20secret

Internet of Things Cockpit: Certificate secret

You might ask: What do I need it for, why do I get a “secret”? A PEM file served by the SAP IoT framework contains two pieces of information:

  1. The SAP server-side certificate which is public. With this certificate all data which we receive from the IoT framework on our device is decrypted and therefore verified. That is, if we trust the certificate – for that the certificate authority (CA) must be known to both parties. However, this is not scope of our setup.
  2. the device’s private key. It is used to encrypt communication to the SAP IoT framework MQTT server. The server holds the public key of the device (as we obtained the .pem-file from the server).

Now here’s the catch: Private keys should be protected against loss. With the key in possession communication can be cloned by any other device and will be authenticated against the server. Therefore, SAP encrypts the private key in the .pem-file and one needs the passphrase (aka: secret) to unlock the key.

Problem for us: The Python MQTT client will ask us anytime for that key it powers up. And it can’t be fed into the program itself (well, it’s a security feature, right?). Fact is, very likely we want to get rid of the passphrase and accept that we have to protect the .pem-file! ?

You need OpenSSL installed for that – it’s three steps. First we extract the private key into a separate file and remove the passphrase protection.

openssl rsa -in <our downloaded>.pem -out private.key -passin pass:<the secret we got>

Then in the second step, we extract the public certificate into a new file.

openssl x509 -inform PEM -in <our downloaded>.pem -outform PEM -out certificate.crt

Lastly, we can combine the two separate files into one.

cat private.key certificate.crt > newcert.pem

Voilà! We got a passphrase-free file for our Python client.

The Python client

The client itself is almost self-explanatory, we need to instantiate it with the alternate ID of the device which is maybe a little special:

client = mqtt.Client(sapIotDeviceID) 

And of course we make use of our new certificate.

import paho.mqtt.client as mqtt
import ssl
import time
import random # to get some 'realistic numbers' :-)

# Parameters
sapIotDeviceID = "<your device alternate ID>"
pemCertFilePath = "./certificates/newcert.pem"
mqttServerUrl = "xyzabc.eu10.cp.iot.sap" # enter the IoT cockpit url here
mqttServerPort = 8883 # Port used by SAP IoT
ackTopicLevel = "ack/" 
measuresTopicLevel = "measures/"

# dummy message
dummyMsg = '{{ "capabilityAlternateId": "<your id>", "sensorAlternateId": "<your sensor id>", "measures": [{{"speed": "{}"}}] }}'

# This function gives a connection response from the server
def onConnect(client, userdata, flags, rc):
    rcList = {
        0: "Connection successful",
        1: "Connection refused - incorrect protocol version",
        2: "Connection refused - invalid client identifier",
        3: "Connection refused - server unavailable",
        4: "Connection refused - bad username or password",
        5: "Connection refused",
    }
    print(rcList.get(rc, "Unknown server connection return code {}.".format(rc)))

# The callback for when a PUBLISH message is received from the server.
def onMessage(client, userdata, msg):
    print(msg.topic + " " + str(msg.payload))

# Send message to SAP MQTT Server
def sendMessage(client, deviceID, messageContentJson):
    time.sleep(random.randint(0,2)) #Wait some random time to make it look realistic
    client.publish(deviceID, messageContentJson)

print("Starting up...")
client = mqtt.Client(sapIotDeviceID) 
client.on_connect = onConnect
client.on_message = onMessage
client.tls_set(certfile=pemCertFilePath, cert_reqs=ssl.CERT_REQUIRED, tls_version=ssl.PROTOCOL_TLS, ciphers=None)
client.connect(mqttServerUrl, mqttServerPort)
client.subscribe(ackTopicLevel+sapIotDeviceID) #Subscribe to device ack topic (feedback given from SAP IoT MQTT Server)
client.loop_start() #Listening loop start 

for _ in range(10): # We send 10 random values
    sendMessage(client, measuresTopicLevel+sapIotDeviceID, dummyMsg.format(random.randint(10,50)))
time.sleep(2) # wait until we have all feedback messages from the server
client.loop_stop

Closing notes

We looked into building an MQTT client with Python that can speak to SAP Cloud Platform Internet of Things. Please note the setup requires ssl package to be imported.

It’s maybe also worth mentioning that the paho-mqtt package is not enabled for multiprocessing. Which means your code must manage the amount of messages you want to send carefully. There are some who launched a new client with every process.

Let me know if you have any suggestions or experiences how to handle multiprocessing (not threading!). Looking forward to your feedback. ?

Assigned Tags

      4 Comments
      You must be Logged on to comment or reply to a post.
      Author's profile photo erick simas grilo
      erick simas grilo

      Hi Gunter!

       

      First of all, very nice blog! Thank you.

       

      I've been struggling by using TLS and i thought it was the way i was using it, but i am getting the same error when connecting to the server following your approach. The same Server URL works when sending data to sensors in SAP Leonardo IoT cockpit by using PAHO Client provided by Eclipse (as an standalone UI app), but when using it with Python or Java, i keep getting server unavailable. Have you ever had a similar error?

       

      Thank you very much!

      Author's profile photo Gunter Albrecht
      Gunter Albrecht
      Blog Post Author

      Good day Erick,

      I had no issues to log in with above code. It's a bit difficult to provide advise without seeing it run. You mention the return code is 3? I would double-check the url and port number and if nothing helps (debugging the code, checking the pem file for correctness, firewall/ virus scanner on local computer etc.) I'd look into the communication on TCP level with Wireshark to understand if the server responds at all.

      Good luck!

      Author's profile photo erick simas grilo
      erick simas grilo

      Thank you! I forgot about the wireshark option to check what is going down in TCP layer... I'll give it a shot.

       

      Author's profile photo erick simas grilo
      erick simas grilo

      Hi! I managed to find out what was happening a few weeks ago.

       

      My version of openSSL was not working properly with the standard certificate chain load that the SSL context setup Paho employs. I had rewritten it based on another Paho extension provided by SAP (https://github.com/SAP-archive/iot-services-sdk), but i had to extend class mqtt_client to update a few things.

      I hope this can help should anyone face the same connectivity issue!

       

      Regards,

      Erick Grilo.