FAQs for SAP Data Hub, developer edition
This page is used to collect frequently asked questions related to SAP Data Hub, developer edition. We will update it regularly.
Where do I find the SAP Data Hub, developer edition?
You find an overview in this blog post. And you can get started by looking at this tutorial.
Can the developer edition be used with Docker Toolbox?
Yes, you can run the SAP Data Hub, developer edition with Docker Toolbox (although we recommend to use Docker for Windows respectively Docker for Mac, since Docker Toolbox is considered a legacy solution).
- Ensure that the Linux VM used to run Docker is sufficiently sized (at least 2 cores, recommended 4 cores, 8 GB of RAM). You can adjust the sizing via Virtualbox.
- Publishing the services running inside the Docker container to 127.0.0.1 (localhost) does not work. Instead you can publish and access them to the docker-machine ip.
ERROR: could not handle api call, failure reason : execution of scheduler plan failed
If this error occurs while storing data in SAP Vora (during turorial Store sensor data in SAP Vora), check that you have correctly maintained the connection between the Kafka Consumer operator and the ToString Converter operator.
You need to connect the message port of the Kafka Consumer operator to the ininterface port of the ToString Converter operator.
Where do I find logs inside the Docker container?
You find the logs related to SAP Data Hub (and SAP Vora) inside the Docker container in the directory /var/log/vora.
How can I access SAP Data Hub, developer edition from “other” hosts?
You have set up SAP Data Hub, developer edition on a host different from your local computer or inside a VM (via Docker Toolbox). You cannot access SAP Data Hub.
You are running the Docker container for SAP Data Hub, developer edition with docker run -ti –publish 127.0.0.1:8090:8090 –publish 127.0.0.1:8998:8998 –publish 127.0.0.1:9225:9225 –publish 127.0.0.1:50070:50070 –name datahub –hostname datahub –network dev-net datahub run –agree-to-sap-license –hdfs –livy as per this tutorial.
When using this tutorial to set up SAP Data Hub, developer edition you bind the container ports to the loopback IP address (aka “localhost”) to restrict access. Hence you can only access it on the host it is set up on.
- For accessing SAP Data Hub, developer edition in conjunction with Docker Toolbox, look at “Can the developer edition be used with Docker Toolbox?”.
- For accessing SAP Data Hub, developer edition on “another” host (different from your local computer), you can do without binding the container ports to an IP address. For details refer to the README file which you have downloaded with the developer edition. Keep in mind: SAP Data Hub, developer edition is not intended to provide a multi-user environment.
ERROR: No matching manifest for unknown…
You try to use SAP Data Hub, developer edition in conjunction with Docker for Windows and get an error “No matching manifest for unknown…”. Please ensure that you use Linux (and not Windows) containers.
Hello, I was able to successfully setup SAP Data Hub as covered in the tutorial at https://www.sap.com/developer/tutorials/datahub-docker-setup.html and the status loop seems to be running as expected.
However, I am not able to access any of the URLs from step 5 Smoke Test
http://localhost:8090 (SAP Data Hub - Data Pipelines)
http://localhost:9099 (Apache Zeppelin)
http://localhost:9225 (SAP Vora Tools)
http://localhost:50070 (Apache Hadoop User Interface)
I suspect there are some changes I need to make in the Virtual Box network settings.
I did notice that an IP of 172.18.0.2 is created in the Docker containter while observing the status loop so I tried with this IP instead of localhost but still get a "Site Cannot Be Reached - ERR_CONNECTION_REFUSED" error when trying to navigate to the URLs.
I have also tried changing the network adapters one at a time to use Bridged Mode. The default settings created in Virtual Box are:
Adapter 1: NAT
Adapter 2: Host Only Adapter (Virtual Box Host Only Ethernet Adapter #3)
Any advice on what I need to modify so that I am able to access the URL's and start working with Data Hub?
I am using Docker Toolbox since I am running Windows 7.
Then, to access the "Data Hub" use http://192.168.99.100:50001
I hope that will help you.
Hi Chris,
let me know if Ayman's solution works.
Here is what worked for me based on changes to the documented step 4:
1. find the docker-machine ip address:
$ docker-machine ip
192.168.99.100
2. Publish using the docker-machine ip address:
$ docker run -ti --publish 192.168.99.100:8090:8090 --publish 192.168.99.100:9099:9099 --publish 192.168.99.100:9225:9225 --publish 192.168.99.100:50070:50070 --name datahub --hostname datahub --network dev-net datahub run --agree-to-sap-license --hdfs –zeppelin
Now these work:
http://192.168.99.100:8090/
http://192.168.99.100:9099/#/
http://192.168.99.100:9225/web/
http://192.168.99.100:50070/
Regards,
Serge
I ended up getting it running using your process here and the issued IP address. I may either set it to permanent or reconfigure for DHCP. First I need to try and acquire more RAM for my laptop so I can run it without risking a crash. I wanted to prove out I could get it running before investing in hardware upgrades. 🙂
Hello,
This works for me as well. But how could I use the hostname instead of the IP-address of the docker machine? Using the hostname throws the error
docker: invalid publish opts format (should be name=value but got '<hostname>:8090:8090')
Thx, Ingo
I just want to add that the "IP address 192.168.99.100" was assigned by the DHCP server it may vary in another installation of VirtualBox.
So, you can also run the datahub without IP in the command.
here's what I'm using:
$ docker run -ti --publish 50000:9225 --publish 50001:8090 --publish 50002:9099 --publish 50070:50070 --publish 8500:8500 --publish 14000:14000 --name devedition --hostname devedition --net dev-net datahub run --agree-to-sap-license --hdfs --zeppelin
Regards,
Ayman
Thanks all for the suggestions. Note my reply above, I was able to get it running finally.
Hi Thorsten,
Thanks for the tutorial, I was able to run it until the last
step :https://www.sap.com/developer/tutorials/datahub-pipelines-storeinvora.html
where i get the following message :
"Error during init of 'com.sap.vora.hdfsLoader': failed to ping SQL connection with DSN v2://vora:2202/?binary=true: timeout. Please ensure host is up and reachable"
It seems that port 2202 is not working. any hint how to debug this issue?
Thanks
Abdelkader
Hallo Abdelkader,
you can check the status of the Vora services under http://<host/ip address>:8500
if all services are running, then check the firewall on your system (disable it or allow port access)
Ayman
Hi Thorsten,
I have installed SAP Data Hub, developer Edition on my Windows 7 (8GB RAM) laptop using docker toolbox. I have successfully build the docker image of Dev edition, but not able to run docker container based on above datahub image. Please find below error screenshot for your reference.
Hi Deepak,
hard to say what goes wrong there. How much CPUs/RAM have you given to the Linux VM (see also question "Can the developer edition be used with Docker Toolbox?")?
Are you using the newest version of SAP Data Hub (i.e. when did you download it)? And does the problem stay the same even when you try a second time / after restarting your computer?
Cheers
Thorsten
Hi Thorsten,
I am trying to install DH Developer Edition on my desktop using Docker but I am not able to build the image. See the error on the screenshot.
I hope you can help me, Thanks a lot.
David
Hi David,
two questions
1) which operating system / Docker version are you using?
2) do you build the image from inside a corporate (and have you set the necessary proxy in the Dockerfile)?
Cheers
Thorsten
Thorsten I am using Windows 10 Pro with Docker version 18.03.
About the second question I am using my personal computer, I am not inside a corporate and I have set the necessary proxy in the Dockerfile using:
ENV http_proxy=http://myproxy:8080
ENV https_proxy=http://myproxy:8080
Thanks
Hi David,
but you go via a proxy? Have you tried without setting the proxy?
If that does not help, can you somehow (mail?) send me the complete log of the build process? Thanks.
Best regards
Thorsten
Thorsten I think you can't post your mail here but could you send me a mail to df.boada914@gmail.com so I can send you the complete log of the build process
Thank you very much.
Hi,
I downloaded the latest version 1.2 but I'm getting an error using the following command:
docker build --tag datahub_1.2 --build-arg VORA_USERNAME=vora --build-arg VORA_PASSWORD=******** .
Kindly advise
cheers
G.
ok, I solved it on my own.
I changed the file dev-edition-helper.sh in the directory DatahubDevEdition/files and replaced the existing line for the LIVY_URL with some other mirror from https://www.apache.org/dyn/closer.lua/incubator/livy/0.5.0-incubating/livy-0.5.0-incubating-bin.zip
For example:
After that the build runs.
Cheers
Gerd
Hi Gerd,
the mirror had a problem on the last weekend. Cheers
Thorsten
Hi Thorsten,
Thanks for creating such a wonderful blog.
I am stuck at the last step of the blog "Store sensor data in SAP Vora". After executing pipeline, I am getting error " Error during init of 'com.sap.vora.hdfsLoader': failed to ping SQL connection with DSN v2://vora:2202/?binary=true: timeout. Please ensure host is up and reachable".
Kindly advise.
Thanks.
I solved the issue. Increased no of cpu for VM.
Thorsten Schneider
I am trying to create a Docker File in the SAP Data Hub Pipeline Modeler. I have created the folder and gave a name for my Docker File. The Docker File Build is started, but i am recieve an error that the docker image cannot build then “Cannot connect to the Docker daemon at unix:///var/run/docker.sock”.
Do you have a suggestion to slove this issue?
Thanks,
Tatiana
Hi Tatiana,
at the moment you are not able to do this with the developer edition. Normally (i.e. not in the developer edition) we will spin up containers when running pipelines. These will run on Kubernetes and they will be the runtime environment for the operators.
With the (current) developer edition we only deliver a single docker image / container. You are not able to define your own Dockerfiles for new operators. Simply spoken the container which runs the developer edition itself is also the runtime environment for all operators.
If you like to build own operators AND create dedicated Docker files for these, please use our trial edition.
We know that this behavior is not ideal, but it will stay like this at least for 2.3 / 2.4.
Sorry!
Cheers
Thorsten
Thanks for the Information!
Hi.
I'm getting this same error.
Is Data Hub staying like this even with Developer Edition 2.3?
I'm trying to build my own operator and it is not possible.
Thanks.
Hi,
yes. This will stay like this at least for 2.3 and 2.4. You can alternatively look at the SAP Data Hub, trial edition which does not have this restriction.
Best regards
Thorsten
Hi,
If I want to use pandas/numpy/scikit-learn python libraries, I need to first create a docker file in the datahub. If the datahub developer edition does not support creating new docker file, is there any workaroundthat allows me to use these python libraries?
Thanks!
Hobart
Hi Hobart,
you can try to install the libraries directly into the Docker OS, i.e. you exec bash inside the container and do the installation.
Cheers
Thorsten
Hey Thorsten,
after installing everything and getting it to run on Docker for Linux, I integrated a Hadoop Cluster to Read and Write the Data to.
Now I would love to try the Leonardo MLF Client, but when it comes to configuring it with the rigth API’s from the SAP Business Hub I really am not able to make it work.
Any hints or documentation you could refere me to?
Many thanks in advance!
Julian
Hey Julian,
question: I think we have an example graph for MLF (I am not sure, if it is delivered as part of the developer edition... need to check, but will not find the time today).
Will try to look into or ask somebody to look into.
BR
Thorsten
Hey Thorsten,
Thank you for looking into it!
Looking forward to hearing from you
Julian
Running Set up for SAP Data Hub, developer edition 2.4 tutorial and I am getting an error when I run the following docker command per the tutorial:
PS C:\SAP\dh_dev2_4\DatahubDevEdition> docker build --tag datahub .
Sending build context to Docker daemon 1.028GB
Step 1/14 : ARG BASEIMAGE="sapdatahub/dev-edition-base:15.0-01"
Step 2/14 : FROM ${BASEIMAGE}
Get https://registry-1.docker.io/v2/sapdatahub/dev-edition-base/manifests/15.0-01: unauthorized: incorrect username or password
I am logged onto the website https://developers.sap.com/tutorials/datahub-docker-v2-setup.html and in those steps it does not give me a user name and password and the docker command does not ask for one?
Hi Terry,
no imediate idea.... I just tested on my laptop and all works as expected.
You should not need a username / password.
If the problem is still there, can you give some more details about the environment you are using? Windows 10... with Docker for Windows?
What happens when you run "docker pull sapdatahub/dev-edition-base:15.0-01"?
Cheers
Thorsten