Understanding containers (part 03): one-shot containers
Thank for all the feedback — here on the Community, as well as on LinkedIn and Twitter — after the part 02 of this series. Please keep it coming!
Quick recap before we dig into today’s short post:
- a container is a running instance of an image (to simplify for now);
- we can execute different commands from already running container…
…like some client commands.
But what if I want to execute a client without running the whole container with a server, like we’ve done so far executing OrientDB’s console client from within a container running OrientDB server.
Example 1: an MQTT subscriber
Some of you might remember that I’ve done some work with IoT in the past, and even today using IoT as a use case for SAP Data Hub CodeJams. On of the technologies we are using there is a lightweight protocol MQTT.
There are quite a few of public servers with MQTT brokers out there. One of them — http://test.mosquitto.org/bbc/ — has a constant stream of BBC closed captions published in three different MQTT topics.
For our exercise we want to receive a stream from a topic
bbc/subtitles/bbc_two_england, but without installing any client software directly on our computer. Instead we are going to use
eclipse-mosquitto Docker repository.
latest image of it contains
mosquitto_sub — a simple MQTT client that will subscribe to topics and print the messages that it receives.
So, let’s run the following command…
docker run -it --rm eclipse-mosquitto mosquitto_sub -h test.mosquitto.org -t bbc/subtitles/bbc_two_england/#
…and understand the results.
We requested Docker to
run a new container using a (
latest by default) image of
eclipse-mosquitto and to execute the
mosquitto_sub client to connect to the MQTT broker
test.mosquitto.org and to output messages received on the subtopics (therefore
# multi-level wildcard at the end) of
The output within the red arrow is coming from the Docker, while the output within the yellow is what was coming from the stream of closed captions at the time of the live BBC2 broadcast (it was the history of “fairy? lights”).
I had to focus on writing this post instead of following the story of lights, so I pressed
Ctrl+C. The command execution stopped, and Docker not only stopped the container, but as well removed it, because of
--rm option in the
Example 2: Drop-in replacement of local commands
I did not need to install MQTT client software on my laptop, check for any of its dependencies or think how to remove it when I not need it anymore!
What’s more I can create aliases (I am using MacOS with Docker Desktop right now) to create drop-in replacement of these commands on my laptop.
Now in one terminal window let me run
mosquitto_sub as if it would be a client available in my OS…
alias mosquitto_sub='docker run -it --rm eclipse-mosquitto mosquitto_sub' mosquitto_sub -h test.mosquitto.org -t sygyzmundovych
…while in another terminal session let me run
mosquitto_pub connecting to the same MQTT broker and publishing messages to the same topic
sygyzmundovych from my keyboard (option
alias mosquitto_pub='docker run -it --rm eclipse-mosquitto mosquitto_pub' mosquitto_pub -h test.mosquitto.org -t sygyzmundovych -l
Same messages are received and displayed in the first window as typed and published in the second.
If we check running containers we should see two of them based on
Try to stop — using
Ctrl+D (which is
EOF i.e. end-of-input) for
mosquitto_pub — and to run again these
mosquitto pub and sub commands to see results changing in
docker ps output.
Example 3: Connecting to a container’s network
I assume that you have the container
myorientdb01 (from previous posts) running, e.g. by restarting it with
docker start myorientdb01, if it was stopped in the meantime.
Let’s check its hostname with
docker exec myorientdb01 hostname. The hostname of my currently running OrientDB server’s container is
Usually containers are very lightweight, as well in a terms of containing only minimal set of software required to run the container. Even many usual commands, like
ifconfig, you take for granted are missing. This reduces the size of the image, but as well makes troubleshooting harder should things get out of control.
Instead of installing this and other network utilities inside the running OrientDB container we will use another method: a one-shot container with required network tools connecting to running OrientDB container’s network.
There is an image described as “network trouble-shooting swiss-army container” in the public Docker Hub registry: https://hub.docker.com/r/nicolaka/netshoot. That image has a set of powerful tools that can be used to troubleshoot Docker networking issues.
Actually that container is really useful, comparing to that real 85 tools Giant Swiss Army Knife 🙂
ifconfig using that image…
docker run -t --rm nicolaka/netshoot ifconfig
…which will pull the image from the Docker registry, create one-shot container (i.e. automatically removed after execution of
ifconfig command because of
The IP address of that one-shot container was
172.17.0.3, but the finished container got deleted. Obviously the image itself remained locally, and will not be pulled during next run of a container.
Now, let’s find the IP address of the container running OrientDB server…
docker run -t --rm --net container:myorientdb01 nicolaka/netshoot ifconfig
--net container:<containerid> to the
…and the address is
We are not going to spend more time on Docker networks right now, as it requires a separate post or two. But you can still play with other tools included in the
Example 4: Useless, but…
This last example is over-engineering, as we have OrientDB console available in the container with the running DB server. We used it already in the previous post.
But “Repetition is the mother of all learning!“.
This time let’s delete the previously created
mydb database using one-shot container with an OrientDB console connecting to the database running on
172.17.0.2. As you’ve seen above this is the IP address of a
myorientdb01 container tht I have on my laptop right now, but might be different in yours.
docker run -t --rm orientdb bin/console.sh "drop database remote:172.17.0.2/mydb root root"
And the database is gone from that OrientDB server. Make sure you use your proper password of db user
root. In my case the password was
root too, and now I see it might be confusing, and is clearly not a good practice 😉
We will keep digging into Docker and containers in next posts. I will tag these posts UnderstandContainers for easy search.
-Vitaliy (aka @Sygyzmundovych)
Another fine blog on containers, Vitaliy!
I'm really enjoying this series. Thanks!
Hi Doug. It took me quite some time to find an MQTT server with a topic with a constant feed. I thought as well that it should be so obvious to find some sample stream with e.g. temperature or air quality data... but not. If you found something interesting publicly available, then please share it too.
Hi Vitaliy. I've been searching, and the best I've found is the real-time public "high frequency positioning" MQTT feed at mqtt.hsl.fi of transit vehicles from the Finnish Transport Agency. Buses, trains and tram vehicles report their route, speed, long/lat, odometer, start/stop, etc. It's all very well documented here: https://digitransit.fi/en/developers/apis/4-realtime-api/vehicle-positions/
The feed can be overwhelming with such a large topic tree and payload, but parsing it down to a specific train (mode), operator (0090) and train (01021) running at the moment I write this was fun! Here's my train example, using a one-shot mosquitto container client to keep this on topic. Replace "01021" with + to see all active trains running for operator 0090.
This HFP data might make for a good future IoT, geospatial CodeJam pipeline app in Data Hub. I haven't figured out how to consume from mqtt.hsl.fi using the SDH delivered MQTT operator using tcp, though. The broker mqtt.hsl.fi seems to publish in mqtt(s) or ws(s), not tcp. A further constraint in SDH Dev Edition might be access to esri-like geo data for mapping the positioning of the vehicles. Just thinking through how to apply this MQTT source for SDH experiments...
Regardless, I wanted to highlight this excellent public MQTT topic source!
Thank, Douglas Maltby for sharing!
I did request earlier this year to update the documentation and I see different schemes
wsare listed in https://help.sap.com/viewer/97fce0b6d93e490fadec7e7021e9016e/Cloud/en-US/709a1b65b93d4f5f90c992b3442f92ab.html.
I think you should be able to use
wss://mqtt.hsl.fi:443/in the operator as well, even if
wssis not listed.
Excellent! I was just now able to get broker tcp://mqtt.hsl.fi:1883/ to work with the delivered SDH MQTT consumer client (after the pipeline engine restarted itself). I see wss and ssl are supported in Data Intelligence, but not sure if that also applies to SDH 2.4 Dev Ed.
I had been considering trying to install the eclipse-mosquitto in the datahub container to create a new mosquitto client MQTT operator, but ultimately just got broker tcp://mqtt.hsl.fi:1883/ to work with a currently running train (01077) to Lentoasema- Helsinki (the airport). Here's the same via the mosquitto one-shot container
Now to figure out how plot the MQTT feed info on a map in SDH (google maps with http client?) or SAC...or maybe just save the data to hdfs or vora and plot speed over the journey time of the train.