Skip to Content
Technical Articles

Extending SAP Data Intelligence Part 2: Packaging, deploying and running solution containing custom application

Introduction

This post documents the hands-on technical steps I followed during my journey of exploring extensibility of SAP Data Intelligence and is a continuation of my post titled “Extending SAP Data Intelligence Part 1: Configuring a Custom Solution for TensorBoard”. In the first post, I described how to develop the application descriptor and solution manifest file required to create a custom solution. As with the first post, take this as the result of personal experience and not as a “best practice” guide.

Prerequisites

  • Access to an SAP Data Intelligence instance.
    • SAP Data Intelligence (cloud edition) Version: 2006.1.8 was used in this post.
  • You will need to have admin permissions on the SAP DI tenant to install the application and create custom policies.
  • To use Jupyter Notebooks you will need a user that has access to ML Scenario Manager and TensorBoard App

Overview

In this post, I will go through the steps of packaging, deploying a custom solution for SAP Data Intelligence. I also included animated GIFs that I created to visually explain where to go if you are not familiar with the UI.

At the time writing the post I noticed that it was a bit too lengthy to pack into a single post. For that reason, I split it into 2 separate posts to show what I did for my colleagues:

The environment to TensorBoard consists of the following systems and tools:

  • SAP Data Intelligence
    • ML Scenario Manager
    • TensorBoard
    • Tensorflow installed in Jupyter Lab

Create a deployable Solution

  1. Create the Solution directory structure:
    user@home ~ % mkdir -p custom_tensorboard_solution/content/vsystem/{apps,icons}
  2. Create manifest.json:
    user@home ~ % cd custom_tensorboard_solution
    user@home ~ % cat << EOF > manifest.json
    {
        "name": "tensorboard-app",
        "version": "0.0.2",
        "format": "2",
        "dependencies": []
    }
    EOF
    
  3. Create blank application descriptor and copy/paste descriptor from the previous article or download from a GitHub Gist link:
    user@home ~ % touch content/vsystem/apps/tensorboard-app.json
  4. Get an icon for the project:
    user@home ~ % curl https://tensorboard.dev/static/tb_sharing_2.png -o \
      content/vsystem/icons/TensorBoard.png
  5. Display solution structure:
    user@home ~ % tree .                                                      
    .
    ├── content
    │   └── vsystem
    │       ├── apps
    │       │   └── tensorboard-app.json
    │       └── icons
    │           └── TensorBoard.png
    └── manifest.json
    
    4 directories, 3 files
  6. Create a solution package
    user@home ~ % zip -r tensorboard-app.zip manifest.json content/

    Alternatively, if you have the System Management Command-Line Client (vctl) installed then the packaging can be done as follows:

    user@home ~ % vctl solution bundle custom_tensorboard_solution/

Deploy Custom Solution

  1. Find the Solution package tensorboard-app.zip created in the previous step.
  2. Add the Solution to your tenant as described in the official Manage Strategies documentation. The steps are:
    1. Log in as tenant administrator on your SAP Data Intelligence tenant.
      Log%20in%20as%20tenant%20admin
    2. Open the System Management application.
    3. Go to Tenant, select Solutions, and click +.
    4. Choose the tensorboard-app.zip file in the file dialog and confirm.
    5. Go to the Strategies section and click Edit Pencil.
      Add the newly added tensorboard-app to the strategy and click save.
      NOTE: In case you are not able to edit the strategy (edit button does not exist) you need to login to the SAP Data Intelligence system tenant and assign the solution to the strategy there.
      Add%20Solution%20via%20System%20Manager
    6. The TensorBoard application will now show up in the launchpad of all users who have rights to start any application.

Create custom policies

  1. Create the policies and user in your tenant as described in the official Manage Policies and Manage Users documentation. The steps to do this are:
    1. Log in as tenant administrator on your SAP DI tenant.
    2. Open the Policy Management application.
    3. Create a new policy by clicking on +.
    4. Enter custom.tensorboard.start as the Policy Id and deselect the Exposed option. Add a new resource by clicking + and enter the app name tensorboard-app, click Okay and then Create.
    5. Create another policy by clicking on + once again.
    6. Enter custom.developer as the Policy Id, add the policy created in step 4 and click Create.
    7. Navigate to the System Management application.
    8. Go to Users, create a new user or select an existing user.
    9. Click on + and select the role created in step 6 and click Assign.

The user will now have access to start an instance of the TensorBoard application

 

Create a scenario in ML Scenario Manager

  1. Log in as an ML User on your SAP Data Intelligence tenant.
  2. Open the ML Scenario Manager application.
  3. Click Create, enter TensorBoard Sample in name and click Create.
  4. Navigate to the Notebooks section and click Create. In the dialog window, enter TensorBoard Test in name and click Create.
  5. Launch a terminal from the file menu and install TensorFlow by entering pip install tensorflow==1.15 in the terminal and pressing enter.
  6. Add sample code to notebook TensorBoard Test run each cell and click save icon.
    NOTE: I used a tutorial for TensorBoard code from this link to test if the app is working.

    # Import TensorFlow and show version
    import tensorflow as tf
    tf.__version__​
    import numpy as np
    
    X_train = (np.random.sample((10000,5)))
    y_train =  (np.random.sample((10000,1)))
    X_train.shape
    
    feature_columns = [
          tf.feature_column.numeric_column('x', shape=X_train.shape[1:])]
    DNN_reg = tf.estimator.DNNRegressor(feature_columns=feature_columns,
    # Indicate where to store the log file    
         model_dir='/vhome/tf/train/linreg',    
         hidden_units=[500, 300],    
         optimizer=tf.train.ProximalAdagradOptimizer(      
              learning_rate=0.1,      
              l1_regularization_strength=0.001    
          )
    )
    
    # Train the estimator
    train_input = tf.estimator.inputs.numpy_input_fn(    
         x={"x": X_train},    
         y=y_train, shuffle=False,num_epochs=None)
    DNN_reg.train(train_input,steps=3000)

  7. Click on the Quick Launch navigation for TensorBoard and see written logs.
    TensorBoard%20Running%20from%20SAP%20DI

 

Closing and further reading

It was a great experience exploring how to extend SAP Data Intelligence with custom solutions and applications. Creating and deploying is really easy once you understand the components and the steps in creating a Solution.

I hope you enjoyed this blog post and that it gives you a general understanding of components that are part of a custom solution and how to deploy a solution in SAP Data Intelligence. Let me know if you have questions or feedback in the comments section.

If you want more information about everything SAP Data Intelligence related I recommend the following:

6 Comments
You must be Logged on to comment or reply to a post.
  • Hello Lochner,

     

    Thanks for the great read! Very informative.

    I was wondering is the same possible to host a custom ReactJS app running on Node inside DI?

    • Hi Denish,

       

      That is possible although I haven't tested it myself and will have to go through the changelogs to note any new restrictions on custom apps.

       

      An easy way to do this is to create a DockerFile that encapsulates the app. (look at link for example of a ReactJS app that is Dockerized with NodeJS). You can then publish DockerFile on a public repo and then follow the setup from my blog post in part 1.

       

      You can technically deploy anything that can run in Kubernetes on DI (with trial and error). The application descriptor file is essentially a Kubernetes deployment file.

      • Thank you for the reply Lochner!

        Trial and Error is what I was doing till now :), will try publishing the DockerFile and then try and refer it in the deployment JSON file.

        Will keep you posted and if it works.

        Cheers and stay safe,

        Dinesh

      • Hi Lochner Louw

        I did try somethings out but with no luck. It would help us a lot if you could check this once and if it rings any familiar bells.

        1. Created a docker image for the react app
        2. Published the image onto docker hub as a publicly available image (Refer here for the image)
        3. Created a solution in similar lines to your tutorial's Part 1.

        When I installed this solution in DI, I did get a new tile in DI for the app but when I launched it, a container would not be assigned to it, and the app failed with the below error,

        unable to get route "react-app": error starting app: timeout waiting for app to be ready. App is in state Pending. Error reason: Container is in state waiting
        I am new to Kubernetes so I just kinda replicated what you have in your application descriptor, except to use the public image that I have created.
        Is there a way to get access to the startup logs of the kubernetes containers in DI so that I can better understand what's happening during the startup.
        Thanks,
        Dinesh
  • This is the react-app.json descriptor file I am using

     

    {
      "name": "ReactUIEnabler",
      "type": "kubernetes",
      "apiVersion": "v2",
      "version": "0.0.2",
      "icon": "/vsystem/icons/react.png",
      "mounts": { "vhome": true },
      "body": {
        "kubernetes": {
          "apiVersion": "v2",
          "service": {
            "spec": {
              "ports": [
                {
                  "port": 3333
                }
              ]
            }
          },
          "deployment": {
            "spec": {
              "template": {
                "spec": {
                  "containers": [
                    {
                      "name": "react-app",
                      "image": "kmdee6/react-apps:latest",
                      "command": [
                        "npm start"
                      ],
                      "args": [
                        "--logdir",
                        "/vhome/node",
                        "--host",
                        "0.0.0.0"
                      ],
                      "resources": {
                        "requests": {
                          "cpu": "0.1",
                          "memory": "2Gi"
                        },
                        "limits": {
                          "cpu": "0.5",
                          "memory": "5Gi"
                        }
                      },
                      "securityContext": {
                        "runAsUser": 1000
                      },
                      "ports": [
                        {
                          "containerPort": 3333
                        }
                      ]
                    }
                  ]
                }
              }
            }
          }
        }
      }
    }
    • Hi Dinesh,

      I took a look at your docker and compared the app descriptor file.

      You need to do the following if you want to use the image you currently pushed:

      1. Leave the "args" section empty ("args": [],) in the descriptor - What I specified is only for TensorFlow
      2. Change the ports in the descriptor file to 80 - that is what port Nginx is running on in the docker container (that is port which is specified in the DockerFile as EXPOSE 80)

      I don't see npm installed in the container and I am not sure where you put the react app. Take a look at this link on React and Kubernetes before attempting it on DI.

      Make sure you get the docker container working locally before trying again. This will mean that you need to bundle everything into a docker image. I would recommend looking more into Docker and Kubernetes and also check out SAP Data Intelligence: Git Workflow and CI/CD Process post by Christian Sengstock. The post links to a git repo that contains an example for running Code Server in DI. This will be more helpful on what the DockerFIle looks like.