Create custom Jupyter Lab Python kernels on SAP Data Intelligence

[LarissaH]

In this tutorial, I will show you how to create additional python kernels on the SAP Data Intelligence to be used within the Jupyter Lab and the ML Scenario Manager.

I, like many of you probably, really like working with the SAP Data Intelligence. What I probably use most, is the Jupyter Lab application, which can be activated and used just like any other known notebook server. Here, you can access your datasets and data sources, you can visualize data and try out code, and produce first results.

But maybe you will be in the same situation as I am: You are working on several projects that require different versions of python libraries, or their dependencies require different versions. You need to control exactly, what you have installed and what is not installed, so this gets pretty messy in one single environment.

The solution: Use different python kernels for different use cases / projects!

When you open a Jupyter notebook in the Jupyter Lab, you will see the label "Python 3" on the top right corner. This indicates on which kernel your notebook is currently running. If you click on it, a small popup window will appear, giving you the option to switch between different kernels. But: There are no other proper kernels (except for kernels from other sessions, which will not be permanent).

So, let's start. Let's assume that you want to use LightGBM on the SAP Data Intelligence. Currently, it is not installed.

Step 1: Open a terminal and activate conda.

Python and python libraries on the SAP Data Intelligence are managed with conda (instead of pip, which might be known to a broader audience). When you use conda on the DI for the first time, you will have to initialize it, simply typing:

conda init bash

Or any other shell option that you want to use. It might be the case that you have to re-load the terminal, that you are using. In this case, simply close the terminal tab and open a new one. Be sure that now the environment has loaded, this is indicated by a name in parenthesis at the beginning of each new terminal line. Most probably, it is called "base":

(base) labuser@jupyter-<some-cryptical-numbers>:~>

If this is not the case, you can activate the environment by typing

conda activate <env-name>

Otherwise, if you want to start completely new, you can create your own environment with the following command and activate it later on:

conda create --name <env-name> python=3.6

In case of questions or problems, please also refer to the official and extensive conda documentation.

Step 2: Install wanted libraries.

This is really simple, as you can just install all the libraries you want to use via conda, for example like this:

conda install lightgbm

Step 3: Install and activate ipython for your kernel.

You need to install ipykernel inside of your environment, in order to connect it to the Jupyter Lab in the next step. This can be done by typing:

conda install ipykernel

python3 -m ipykernel install --user --name <env-name> --display-name "Python 3 (<env-name>)"

In the last line, you need to replace <env-name> with the name of the environment that you installed your wanted libraries in, so for example "base" or your environment name, if you created one by yourself beforehand.

Step 4: Reload Jupyter Lab.

Now you can reload your Jupyter Lab page and review, if the new environment is there for selection.

Be aware: If your Jupyter Lab instance is reloaded, you will lose all environments you just created and everything will be reset to default.

I hope, that this tutorial helped you proceed with the SAP Data Intelligence and opened the door for new possibilities. Let me know if you have any questions!