In my recent post Quickly load Covid-19 data with hana_ml and see with DBeaver I used the Jupyter notebook to connect to my SAP HANA database instance in the public cloud via its IP address. But what if I want to connect from a Jupyter notebook running in one container to the OrientDB server running in another container?
If you've been following previous parts of this Understand Containers series, then you should have these two containers on your computer
To get something meaningful we need some data in the OrientDB database and a Python client in the Jupyter environment to connect to that database.
We need an OrientDB driver for Python installed in my Jupyter container from https://github.com/orientechnologies/pyorient.
[At the time of writing this post the proper version of the driver is not available at PyPI, therefore we need to install it from the source with
Ok, the driver has been installed in the Jupyter container.
We need some data, and -- because it is Friday afternoon here already -- what can be the better data to look at, if not at OpenBeer dataset?
If you haven't played with it yet by following some of the tutorials or just by importing it as one of the sample databases in OrientDB Studio, then I prepared this one-liner to import it into the DB server running on
This command is certainly overengineered 🙂 as the same can be done in a less confusing way. But I did want to play with it a bit, and that's the result. Some notes:
Once executed successfully...
...you should see this database in the OrientDB Studio.
Once logged you should be able to run a query like below. This one is to find from the graph what the categories of beers brewed in the United States are (at the time the dataset was created ;-)) ordered by a number of beers linked to each category.
That was just the preparation for what we really want to practice today: two containers (or rather applications inside of them) talking to each other.
Basically, we want to get a result of the same query above, but in Jupyter (running in one container) connecting to the OrientDB (running in another container).
Docker has a few networks preconfigured as can be seen using
Each container by default is assigned to the Docker network
So, in my environment, the IP addresses are:
Let's open a new notebook
We imported
Next, we just run a query and output its result.
It is not a good practice to communicate using IP addresses, because it might change. To solve this and to switch to hostnames we should use Docker's user-defined networks.
Let's create our own
...and connect our two containers to this network.
Note that in the
What is cool, is that in the user-defined network container names are as well hostnames! Using example 3 from Understanding containers (part 03): one-shot containers we can use
Let's try from the Jupyter notebook now changing the IP address with the hostname.
Everything works fine.
But using the container name as a hostname might still not be the best solution, in case we got another container with another name running our OrientDB server and serving data.
In that case, it is better to connect a container to a user-defined network using an alias. So, let's disconnect and reconnect our container
...and update the code in the Jupyter notebook.
?
We used quite a few new concepts today: not just Docker's networks, but executing multiple commands in the container using
Just like you, I am thirsty for more. But it is Friday evening, so let me quench my other thirst with ... another beer. Repeating after Ray Bradbury "Beer's intellectual..."
Cheers ?
-Vitaliy (aka @Sygyzmundovych)
Some additional setup
If you've been following previous parts of this Understand Containers series, then you should have these two containers on your computer
myjupyter01 and myorientdb01.To get something meaningful we need some data in the OrientDB database and a Python client in the Jupyter environment to connect to that database.
pyorient driver
We need an OrientDB driver for Python installed in my Jupyter container from https://github.com/orientechnologies/pyorient.
docker exec myjupyter01 pip install git+https://github.com/orientechnologies/pyorient[At the time of writing this post the proper version of the driver is not available at PyPI, therefore we need to install it from the source with
pip install git+https://github.com/orientechnologies/pyorient]Ok, the driver has been installed in the Jupyter container.
OpenBeer database
We need some data, and -- because it is Friday afternoon here already -- what can be the better data to look at, if not at OpenBeer dataset?
If you haven't played with it yet by following some of the tutorials or just by importing it as one of the sample databases in OrientDB Studio, then I prepared this one-liner to import it into the DB server running on
orientdb01 container. (It was on macOS, and might not work the same way on Windows host)BEERDIR=/orientdb/databases/OpenBeer sh -c 'docker exec myorientdb01 sh -c "mkdir -p $BEERDIR && cd $BEERDIR && wget -O- http://orientdb.com/public-databases/OpenBeer.zip | jar xv && ls $BEERDIR | wc -l"'This command is certainly overengineered 🙂 as the same can be done in a less confusing way. But I did want to play with it a bit, and that's the result. Some notes:
- The database folder has to be in
/orientdb/databases/, - I wanted to chain a few commands to be executed in the container in one line, so the only way to do it is via
docker exec myorientdb01 sh -c "CMD1 && CMD2", - I wanted to pass the environment variable
$BEERDIRto the commands, and the only way I found is by wrapping thedockercommand into anothersh -c 'CMD0', - The downloadable sample database is a
zipfile, and -- because there is nounzipavailable in the container -- we do the trick using availablejar xto uncompress a file coming fromwgetviastdinstream.
Once executed successfully...
...you should see this database in the OrientDB Studio.
admin / admin are default user and password for this sample db.Once logged you should be able to run a query like below. This one is to find from the graph what the categories of beers brewed in the United States are (at the time the dataset was created ;-)) ordered by a number of beers linked to each category.
select CatName, count(*) as CatCount
from
(select in(HasBrewery).out(HasCategory).cat_name as CatName
from Brewery
where country='United States'
unwind CatName)
where CatName is not null
group by CatName
order by CatCount descThat was just the preparation for what we really want to practice today: two containers (or rather applications inside of them) talking to each other.
Containers talking to each other...
Basically, we want to get a result of the same query above, but in Jupyter (running in one container) connecting to the OrientDB (running in another container).
...using IP
Docker has a few networks preconfigured as can be seen using
docker network ls.Each container by default is assigned to the Docker network
bridge and gets an IP address in that network. We can find these IP addresses by scrolling through the output of the docker network inspect bridge. Or we can use the --format option to output only what we need using a Go template.I dunno about you, but it is the first time for me using Go templates. It took me a while to get this syntax working in the command below. I'll be grateful if you can share some improvements in comments!
docker network inspect bridge --format '{{range .Containers}}{{.Name}} {{.IPv4Address}}; {{end}}'So, in my environment, the IP addresses are:
172.17.0.2formyorientdb01,172.17.0.3formyjupyter01.
Let's open a new notebook
OrientDB_OpenBeersDB in the Jupyter and try!We imported
pyorient and then used it to open OpenBeer db with the driver's client. This client connected to server on 172.0.0.2 port 2424. This port is used by OrientDB for clients that support Network Binary Protocol, like pyorient.Please note that we use the port from inside the container, not the one to which it might be mapped on the host running Docker!
Next, we just run a query and output its result.
import pyorient
#
client = pyorient.OrientDB("172.17.0.2", 2424)
odbclusters=client.db_open("OpenBeer", "admin", "admin")
#
categories = client.query(
"select CatName, count(*) as CatCount from (select in(HasBrewery).out(HasCategory).cat_name as CatName from Brewery where country='United States' unwind CatName) where CatName is not null group by CatName order by CatCount desc"
)
#
for category in categories:
print(category.oRecordData)...using a hostname
It is not a good practice to communicate using IP addresses, because it might change. To solve this and to switch to hostnames we should use Docker's user-defined networks.
Let's create our own
mynet01 network...docker network create mynet01
docker network ls...and connect our two containers to this network.
docker network connect mynet01 myorientdb01
docker network connect mynet01 myjupyter01
docker network inspect mynet01 --format '{{range .Containers}}{{.Name}} {{.IPv4Address}}; {{end}}'Note that in the
mynet01 our containers have IP addresses in the range 172.18.0.0/16, while in the default bridge network they have IP addresses are in the range 172.17.0.0./16.What is cool, is that in the user-defined network container names are as well hostnames! Using example 3 from Understanding containers (part 03): one-shot containers we can use
ping to check how it is working.docker exec myorientdb01 ping myjupyter01
docker run -t --rm --net container:myorientdb01 nicolaka/netshoot ping myjupyter01Let's try from the Jupyter notebook now changing the IP address with the hostname.
client = pyorient.OrientDB("myorientdb01", 2424)Everything works fine.
...using an alias
But using the container name as a hostname might still not be the best solution, in case we got another container with another name running our OrientDB server and serving data.
In that case, it is better to connect a container to a user-defined network using an alias. So, let's disconnect and reconnect our container
myorientdb01 to mynet01 network using --alias...docker network disconnect mynet01 myorientdb01
docker network connect --alias myorientdb mynet01 myorientdb01...and update the code in the Jupyter notebook.
client = pyorient.OrientDB("myorientdb", 2424)?
We used quite a few new concepts today: not just Docker's networks, but executing multiple commands in the container using
sh -c "...&&..." and using Go templates to format the command output too.Just like you, I am thirsty for more. But it is Friday evening, so let me quench my other thirst with ... another beer. Repeating after Ray Bradbury "Beer's intellectual..."
Cheers ?
-Vitaliy (aka @Sygyzmundovych)
- SAP Managed Tags:
1 Comment
