Technical Articles
Understanding containers (part 03): one-shot containers
Thank for all the feedback — here on the Community, as well as on LinkedIn and Twitter — after the part 02 of this series. Please keep it coming!
Quick recap before we dig into today’s short post:
- a container is a running instance of an image (to simplify for now);
- we can execute different commands from already running container…
…like some client commands.
But what if I want to execute a client without running the whole container with a server, like we’ve done so far executing OrientDB’s console client from within a container running OrientDB server.
Example 1: an MQTT subscriber
Some of you might remember that I’ve done some work with IoT in the past, and even today using IoT as a use case for SAP Data Hub CodeJams. On of the technologies we are using there is a lightweight protocol MQTT.
There are quite a few of public servers with MQTT brokers out there. One of them — http://test.mosquitto.org/bbc/ — has a constant stream of BBC closed captions published in three different MQTT topics.
For our exercise we want to receive a stream from a topic bbc/subtitles/bbc_two_england
, but without installing any client software directly on our computer. Instead we are going to use eclipse-mosquitto
Docker repository.
The latest
image of it contains mosquitto_sub
— a simple MQTT client that will subscribe to topics and print the messages that it receives.
So, let’s run the following command…
docker run -it --rm eclipse-mosquitto mosquitto_sub -h test.mosquitto.org -t bbc/subtitles/bbc_two_england/#
…and understand the results.
We requested Docker to run
a new container using a (latest
by default) image of eclipse-mosquitto
and to execute the mosquitto_sub
client to connect to the MQTT broker test.mosquitto.org
and to output messages received on the subtopics (therefore #
multi-level wildcard at the end) of bbc/subtitles/bbc_two_england
topic.
The output within the red arrow is coming from the Docker, while the output within the yellow is what was coming from the stream of closed captions at the time of the live BBC2 broadcast (it was the history of “fairy? lights”).
I had to focus on writing this post instead of following the story of lights, so I pressed Ctrl+C
. The command execution stopped, and Docker not only stopped the container, but as well removed it, because of --rm
option in the run
command.
Example 2: Drop-in replacement of local commands
I did not need to install MQTT client software on my laptop, check for any of its dependencies or think how to remove it when I not need it anymore!
What’s more I can create aliases (I am using MacOS with Docker Desktop right now) to create drop-in replacement of these commands on my laptop.
Now in one terminal window let me run mosquitto_sub
as if it would be a client available in my OS…
alias mosquitto_sub='docker run -it --rm eclipse-mosquitto mosquitto_sub'
mosquitto_sub -h test.mosquitto.org -t sygyzmundovych
…while in another terminal session let me run mosquitto_pub
connecting to the same MQTT broker and publishing messages to the same topic sygyzmundovych
from my keyboard (option -l
).
alias mosquitto_pub='docker run -it --rm eclipse-mosquitto mosquitto_pub'
mosquitto_pub -h test.mosquitto.org -t sygyzmundovych -l
Same messages are received and displayed in the first window as typed and published in the second.
If we check running containers we should see two of them based on eclipse-mosquitto
running.
Try to stop — using Ctrl+C
for mosquitto_sub
and Ctrl+D
(which is EOF
i.e. end-of-input) for mosquitto_pub
— and to run again these mosquitto
pub and sub commands to see results changing in docker ps
output.
Example 3: Connecting to a container’s network
I assume that you have the container myorientdb01
(from previous posts) running, e.g. by restarting it with docker start myorientdb01
, if it was stopped in the meantime.
Let’s check its hostname with docker exec myorientdb01 hostname
. The hostname of my currently running OrientDB server’s container is 99fe8d28d3c3
.
Usually containers are very lightweight, as well in a terms of containing only minimal set of software required to run the container. Even many usual commands, like ifconfig
, you take for granted are missing. This reduces the size of the image, but as well makes troubleshooting harder should things get out of control.
Instead of installing this and other network utilities inside the running OrientDB container we will use another method: a one-shot container with required network tools connecting to running OrientDB container’s network.
There is an image described as “network trouble-shooting swiss-army container” in the public Docker Hub registry: https://hub.docker.com/r/nicolaka/netshoot. That image has a set of powerful tools that can be used to troubleshoot Docker networking issues.
Actually that container is really useful, comparing to that real 85 tools Giant Swiss Army Knife 🙂
Let’s run ifconfig
using that image…
docker run -t --rm nicolaka/netshoot ifconfig
…which will pull the image from the Docker registry, create one-shot container (i.e. automatically removed after execution of ifconfig
command because of --rm
option).
The IP address of that one-shot container was 172.17.0.3
, but the finished container got deleted. Obviously the image itself remained locally, and will not be pulled during next run of a container.
Now, let’s find the IP address of the container running OrientDB server…
docker run -t --rm --net container:myorientdb01 nicolaka/netshoot ifconfig
…by adding --net container:<containerid>
to the run
command…
…and the address is 172.17.0.2
.
We are not going to spend more time on Docker networks right now, as it requires a separate post or two. But you can still play with other tools included in the netshoot
image.
Example 4: Useless, but…
This last example is over-engineering, as we have OrientDB console available in the container with the running DB server. We used it already in the previous post.
But “Repetition is the mother of all learning!“.
This time let’s delete the previously created mydb
database using one-shot container with an OrientDB console connecting to the database running on 172.17.0.2
. As you’ve seen above this is the IP address of a myorientdb01
container tht I have on my laptop right now, but might be different in yours.
docker run -t --rm orientdb bin/console.sh "drop database remote:172.17.0.2/mydb root root"
And the database is gone from that OrientDB server. Make sure you use your proper password of db user root
. In my case the password was root
too, and now I see it might be confusing, and is clearly not a good practice 😉
We will keep digging into Docker and containers in next posts. I will tag these posts UnderstandContainers for easy search.
Stay tuned!
-Vitaliy (aka @Sygyzmundovych)
Another fine blog on containers, Vitaliy!
I'm really enjoying this series. Thanks!
Doug
Hi Doug. It took me quite some time to find an MQTT server with a topic with a constant feed. I thought as well that it should be so obvious to find some sample stream with e.g. temperature or air quality data... but not. If you found something interesting publicly available, then please share it too.
Cheers!
Hi Vitaliy. I've been searching, and the best I've found is the real-time public "high frequency positioning" MQTT feed at mqtt.hsl.fi of transit vehicles from the Finnish Transport Agency. Buses, trains and tram vehicles report their route, speed, long/lat, odometer, start/stop, etc. It's all very well documented here: https://digitransit.fi/en/developers/apis/4-realtime-api/vehicle-positions/
The feed can be overwhelming with such a large topic tree and payload, but parsing it down to a specific train (mode), operator (0090) and train (01021) running at the moment I write this was fun! Here's my train example, using a one-shot mosquitto container client to keep this on topic. Replace "01021" with + to see all active trains running for operator 0090.
This HFP data might make for a good future IoT, geospatial CodeJam pipeline app in Data Hub. I haven't figured out how to consume from mqtt.hsl.fi using the SDH delivered MQTT operator using tcp, though. The broker mqtt.hsl.fi seems to publish in mqtt(s) or ws(s), not tcp. A further constraint in SDH Dev Edition might be access to esri-like geo data for mapping the positioning of the vehicles. Just thinking through how to apply this MQTT source for SDH experiments...
Regardless, I wanted to highlight this excellent public MQTT topic source!
Doug
Thank, Douglas Maltby for sharing!
I did request earlier this year to update the documentation and I see different schemes
tcp
,ssl
andws
are listed in https://help.sap.com/viewer/97fce0b6d93e490fadec7e7021e9016e/Cloud/en-US/709a1b65b93d4f5f90c992b3442f92ab.html.I think you should be able to use
wss://mqtt.hsl.fi:443/
in the operator as well, even ifwss
is not listed.Excellent! I was just now able to get broker tcp://mqtt.hsl.fi:1883/ to work with the delivered SDH MQTT consumer client (after the pipeline engine restarted itself). I see wss and ssl are supported in Data Intelligence, but not sure if that also applies to SDH 2.4 Dev Ed.
I had been considering trying to install the eclipse-mosquitto in the datahub container to create a new mosquitto client MQTT operator, but ultimately just got broker tcp://mqtt.hsl.fi:1883/ to work with a currently running train (01077) to Lentoasema- Helsinki (the airport). Here's the same via the mosquitto one-shot container
Now to figure out how plot the MQTT feed info on a map in SDH (google maps with http client?) or SAC...or maybe just save the data to hdfs or vora and plot speed over the journey time of the train.