Rapid Data Growth Requires a Decoupled Approach

What is Enterprise Messaging and Why is it Important?

Publish/Subscribe systems have been around since the late 80s[1], and popular protocols like MQTT since 1999[2]. But the reason they are becoming more prominent in PaaS architectures and the cloud is because of macro digital transformation trends such as hyper-connectivity, big-data, IOT and hyper-scale computing. These macro-trends mean there is a lot more information being generated, passed around and ultimately needing to be processed.

The total amount of data in the world was 4.4 zettabytes in 2013. That is set to rise steeply to 44 zettabytes by 2020. To put that in perspective, one zettabyte is equivalent to 44 trillion gigabytes.[3]

Of course a lot of this cited data is being generated in consumer formats such as video, photos and social media, but some of this data is being generated in enterprise centric scenarios. Cisco in a recent Cloud Index report estimates that IOT data will start generating 400 zettabytes per year by 2018.[4] Cisco cites that use cases such as a Boeing 787 aircraft, which generates 40TB per hour of flight, or a Rio Tinto mining operation that can generate up to 2.4TB of data a minute as examples of why the business world is generating so much data.

And why is there so much demand for these new data sources? Well data is viewed by a lot of enterprises as a goldmine for extracting insights, to enable improved decision making and to facilitate a structured responsive and agile operation. IOT also enables companies to integrate their physical and digital worlds whether they are operating in B2B, B2C or B2X. With more data and real-time (in some cases predictive) analytics companies can make faster more accurate decisions, simplify processes and ultimately drive additional revenues or reduce costs. Companies are also being forced into these areas of innovation due to new market entrants and disruptive competitors. Companies are also seeking to scale their operations while also delivering personalized or unique experiences.

How does Enterprise Messaging Work?

Pub/Sub systems allow for communicating systems to be loosely coupled. Typically, publishers post messages to an intermediary broker who in turn filter and then stores (possibly prioritizing in a queue) and forwards messages to the subscribing system. This decoupled approach makes it easier for programs to communicate with each other, since the only thing each environment needs to understand is the message format and protocol. This ultimately means that companies that leverage a pub/sub architecture are more likely to enjoy a scalable robust communication layer that ensures disparate systems are kept in sync and enable the accurate and timely processing of unprecedented data volumes being received (or sent) from both internal and external systems.

With messaging there are a choice of protocols to pick from. The above diagram seeks to simply state when you should consider which protocol (what is the underlying purpose).[5]

DDS can be implemented with or without message broker since it has a sophisticated quality of service (ability to guarantee delivery). It’s data-centric (as opposed to message-centric and is an extremely scalable, industrial-strength system. Security elements though are in beta. DDS is typically used where outages are unacceptable, performance is measured in millisecodns and scaling of applications or data values is relevant.

MQTT is primarily used in IOT scenarios where an extremely small footprint is needed and telecommunication expenses need to be minimized. Communication is over TCP/IP and can support at most once, at least one and exactly once (pub/sub). Typically, a large number of devices are involved and there is minimal peer to peer interaction. In summary a good protocol for data collection.

CoAP is also popular for small device use cases only this time REST (HTTP) is used. With CoAP delivery is not guaranteed but you can mark messages with confirmation request (request/response). Messages can be uni or multi-cast. Similar use cases to MQTT but where guaranteed delivery is not required.

AMQP is for server-to-server pub/sub messaging. It supports sophisticated credit-based flow control to avoid bottlenecks, queues, transactions, multiplexing to get messages through firewalls, trusted entity authentication via Kerberos. It has its own link and transport layers above TCP/IP. Messages are layered. Typical use cases for AMQP are for specific data transfers between systems (sharing work) and where data loss is unacceptable.

XMPP is an XML based protocol. Data can be sent via persistent XML or streams via TCP connections. Messaging patterns can be pub/sub (via extensions) or point to point. Similar to CoAP there is no guaranteed delivery but is possible via extension. XMPP excels where there are minimal connectivity points and speed/CPU usage are not important. One typical use case is “always on”

Messaging Best Practices

Message brokers can deliver large messages, but in reality, transporting gigabytes of data in a single message is not the most efficient approach to messaging and can have side effects like connection controls being triggered, blocking other clients, exhausting resources, and so on. So make sure when you are designing data coming from your publishing that the underlying messages are small.

Messaging infrastructures are designed to connect multiple systems by exchanging messages between them asynchronously. The main purpose is to send messages as efficiently as possible between senders and consumers in different systems. If you want to store data, use a database, not a queue.

How can you use Enterprise Messaging?

SAP Cloud Platform Enterprise Messaging can scale to millions of messages per second in real-time, you can send and receive messages reliably using open standards and protocols. Thanks to seamless integration with SAP HANA Platform in the cloud and on-premise, you can insert messages directly in SAP HANA database for storage and analyses. You can also decouple application logic, develop micro services, and support event-driven architectures.

• Data to be sent and received between distributed systems.
• Applications on different technology platforms to exchange data.
• Better application performance because the task of message delivery is delegated to a dedicated message broker.
• Data to be temporarily persisted in queues until messages have been delivered reliably to receiving clients.
• Asynchronous (non-blocking) communication between senders and receivers.
• Handling of peak data loads.
• Data buffering when consumers are temporarily offline.
• Loose coupling so you can switch seamlessly between architectures when environment changes.

So in summary applications, services, and devices can be connected and communicate with one another through Enterprise Messaging enabling you to manage and scale these connections easily.

Enterprise Messaging provides reliable message delivery so different applications, services, and devices can send messages even when the receiver is temporarily unavailable. The use of open standards and protocols means you can send and receive messages reliably across different technologies, platforms, and clouds.

Enterprise Messaging is engineered to ensure a high throughput and low latency of your data. Thanks to decoupled communication between senders and receivers and the use of asynchronous messaging, Enterprise Messaging can handle virtually any peaks in data traffic and scale to millions of messages per second whenever required. Thanks for your interest in Enterprise Messaging and don’t be shy!

[1] One of the earliest publicly described pub/sub systems was the “news” subsystem of the Isis Toolkit, described at the 1987 Association for Computing Machinery (ACM) Symposium on Operating Systems Principles conference (SOSP ’87), in a paper “Exploiting Virtual Synchrony in Distributed Systems. 123–138”

[2] https://github.com/mqtt/mqtt.github.io/wiki/questions

[3] http://www.northeastern.edu/levelblog/2016/05/13/how-much-data-produced-every-day/

[4] https://www.cisco.com/c/en/us/solutions/collateral/service-provider/global-cloud-index-gci/Cloud_Index_White_Paper.html?CAMPAIGN=GCI+2014&COUNTRY_SITE=us&POSITION=PR&REFERRING_SITE=PR&CREATIVE=PR+to+GCI+WP

[5] http://www.eejournal.com/archives/articles/20150420-protocols/