Why you should use MQTT in IoT projects and why SAP should support it
While designing an IoT device, there are multiple options to connect them. Once of the considerations should be the protocol being used to chat to other devices or to a server.
Previously, I would have easily chosen to make the interface REST based and have done so with Jeenode project a few years ago. But REST is not ideal as it is designed around a simple request/response model. So you ask “did my account balance change” and the response is returned “no it did not”. So you check again a few minutes later, and get the same response. Sound like a silly example? Actually we’ve learned that it is a very real issue that many customers obsessively check throughout the day, as many as 60 times inflicting a load on back-ends that they weren’t designed for.
REST via HTTP also wasn’t really designed to work on mobile networks. HTTP on mobiles is a bit heavy, fragile and slow and drains batteries quickly.
Andy Stanford-Clark and Arlen Nipper invented MQTT to solve a problem they had: how to do reliable messaging over unreliable networks? In an industrial environment with computationally ”challenged” devices, with restricted power due to solar or battery power, on extremely low bandwidth and often brittle RF communications including satellite. It had to work reliably, it had to use very few computational cycles, consume trivial power and could not hog what little bandwidth there was. And it had to be pub/sub so to break the cycle of violence being inflicted by heavy unnecessary workloads and bandwidth consumption due to all the request/response polling-based monitoring & control systems that were in place at that time.
So Arlen & Andy developed a very simple, extremely efficient publish/subscribe reliable messaging protocol and named it MQ Telemetry Transport (MQTT). A protocol that enabled devices to open a connection, keep it open using very little power and receive events or commands with as little as 2 bytes of overhead. A protocol that has the things built in that you need for reliable behavior in an unreliable or intermittently connected wireless environments. Things such as “last will & testament” so all apps know immediately if a client disconnects ungracefully, “retained message” so any user re-connecting immediately gets the very latest business information, etc.
Stephen Nicholas did a fascinating apples to apples comparison of MQTT vs HTTPS on Android, 3G and WiFi which you can read here. The 3G results are quite interesting:
- 93x faster throughput
- 11.89x less battery to send
- 170.9x less battery to receive
- 1/2 as much power to keep connection open
- 8x less network overhead
Initial versions of MQTT have been sending plain information over networks. In V3.1 you can also pass a user name and password with an MQTT packet of the protocol. Encryption across the network can be handled with SSL, independently of the MQTT protocol itself (it is worth noting that SSL is not the lightest of protocols, and does add significant network overhead). Additional security can be added by an application encrypting data that it sends and receives, but this is not something built-in to the protocol, in order to keep it simple and lightweight. These security features are especially meaningful when the IoT thing is running in e.g. a cloud scenario, in which the thing sends and receives data through the public internet.
SAP, eager to also claim its part in the IoT domain, unfortunately has not really shown any signs of offering MQTT on any of its services yet. It is currently being investigated, but nothing concrete has trickled down yet:
@wolf_gregor @jpenninkhof @schneidertho @h_neumann @qmacro MQTT is under evaluation, but not available as roadmap item, yet
— ruinogueira (@ruinogueira) January 26, 2015
It is really a pitty that SAP is not supporting MQTT yet, as this would require a MQTT to HTTP bridge to collect messages from things into the SAP HANA Cloud. As the MQTT protocol is mature and secure enough to be used over the public internet, I think SAP should also support this IoT standard, especially given the fact that SAP wants claim a good chunk of the IoT domain. I could think of hooking up MQTT to e.g. SAP’s HCI, but perhaps this would also be a good business case to get ESP (Event Stream Processing) to run on the HANA Cloud Platform
Very well written and informative thanks.
I went to the IoT conference at SAPLabs prior to TechEd last year, it seems there are quite a few competing protocols in this space, at the conference each of the vendors got up and showed their solutions and talked about which protocol they used and why, for me as a developer there was very little signal amongst the noise, I felt I would need to do research on each and how they related back to a SAP solution. Thanks now i know a little more about MQTT.
You're right, there are a few more contenders. This pages gives a very nice overview of the alternatives and the pros and cons:
IoT Primer: IoT Protocol Wars: MQTT vs CoAP vs XMPP
I guess it would have been fair to mention the other protocols as well, but when I was doing my research, I was looking for a protocol that would survive a cloud scenario. This would mean that on top of some basic requirements such as light-weight and pub/sub, it needed to support SSL, authentication and would preferrably be TCP (not UDP based).
Because of that, I though only MQTT would fit the bill.
Just wanted to let you know that the SAP Plant Connectivity product (bundled with SAP MII and part of our overall SAP Connected Manufacturing strategy) is planned to support MQTT in the 15.1 release that is set for Q4 2015; so later this year. There are some drawbacks to MQTT at the moment when compared to other protocols in that it's completely unstructured so the target system will have to know how to manage the incoming data on it's own. IMHO, this standard has some room for growth but it's an acceptable transport option...that's why you see that PCo has a native SAP ESP streaming destination as this was well structured and easily secured.
Fantastic overview Jan and a very important point you've highlighted. Yep, SAP are dithering again - or maybe just very poor communication from SAP in this regard. Personally feel SAP HCI could probably be first place to provide support for it (by way of an MQTT adapter that could be downported to SAP PI/O on premise) - This in itself could potentially open up a world of opportunities for customers with a big SAP backend footprint - or cloud footprint - or hybrid footprint.
A personal pipe dream of mine is: SAP builds MQTT broker into ESP. Jokes aside, the momentum that MQTT is picking up can't simply be ignored. Comparisons & IoT protocol wars aside, you've got to at least start supporting most popular IoT protocol first and work your way downwards, or is my thinking too simplistic?
What would be your business use case around having an MQTT interface into ESP, now braded as HANA Smart Data Streaming (SDS)? As previously mentioned there are options with IoT Edge and also with SAP Plant Connectivity to translate various protocols and systems to natively push data into SDS so your streaming scenario may already be covered but without the need for MQTT in the middle.
I think that your thinking may be too simplistic in that the structure of the data inside of the MQTT feed is hard-coded to the receiving application and not generic or open standard. This makes it very difficult to say it's the "best" IoT option or even the most popular. I would argue that OPC-UA is more popular for industrial use cases over MQTT which is mainly focused on servicing telemetry based systems. Also, it is my understanding that this protocol is really only a one-way feed and does not offer the option for synchronous queries or commands further limiting scenarios it can cover.
Hi Sam, thanks for highlighting some things I wasn't aware of.
Was thinking out loud with the MQTT interface in SDS. Some of the business use cases I had in mind was related to sensor triggers (on devices / equipment inside the network, not on the edge). This with a backdrop of heavily integrated SAP environments.Sort of an early watch setup. An example, we have a scenario of a highly mechanized warehousing environment (literally robots traversing around the warehouse & tackling task / instructions that are fed to them via interfaces / integration. A delay of a couple of minutes (due to maybe an integration issue - or stock issue - into SAP EWM or SAP ECC) has a severe knock on effect in the Supply chain. It would be nice to be made aware of a slow down in operations (i.e. robot A is waiting inexplicably long at point C twiddling his thumbs) quite quickly through SDS (via MQTT client requests on these robots) before the integration monitoring teams pick it up through monitoring tasks at intervals. Was also thinking about it's possible integration use cases in disparate environments (inside a network). From a systems perspective, used to identify trends of high latency network traffic issues, slow disk I/O, tables filling up too rapidly, high swapping, high CPU usage (triggering possible re-sizing discussions) etc. Sure some aspects could be covered through SAP Solution Manager and Wily Introscope but these diagnosis tools are mostly used retrospectively. My whole premise was around monitoring inside the network without burdening the network with heavy requests.
MQTT has it's limitations like everything else in life :-). But if you throw in an HCI (or SAP PI/O) MQTT adapter (along with SDS detecting event trends if it has an MQTT interface) then you somewhat mitigate the interoperability concerns. If you look at IBM MQ & Apache's ActiveMQ, they both offer native capabilities of automatically converting MQTT messages to JMS compliant messages - any middleware solution can take over from that point as a subscriber. The fire-and-forget pub/sub mechanism is immensely valuable from a landscape wide monitoring perspective. Tx
Thanks for sharing with us this interesting blog. Just an update from the field; from now on it's possible to integrate IoT devices with the SAP HANA Platform (and on-premise SAP modules) using our native MQTT adapter for HANA Cloud Integration. read more about it here https://www.sapappcenter.com/p/20063 and here Rojo - Rojo MQTT Adapter for SAP HANA Cloud Integration. KR, Roberto Viana
Hi Roberto, great to hear that SAP is also embracing MQTT. It's very necessary for many IoT scenario's, and a great step that it is available through HCI now 🙂
Hope I'm not too demanding, but it would make this birthday present complete (it's my birthday today) if SAP were to decide to package this as part of the IoT service as well. Do you think that could happen in the future?
fyi - at TechEd 2016 I will give a presentation on the planned HCP-based Messaging-as-a-Service and also give a 2h hands on (INT267 & INT109).