Zen and the Art of SAP Data Intelligence. Episode 3: vctl, the hidden pearl you must know

When I joined SAP five years ago I became a member of a small team of developers who were hired to obey one single commandment: innovate!  But the initial group was so small compared to the size of their goals that an important corollary was immediately appended: automate everything!  The zeal with which the team followed the directive made it possible to quickly grow and deliver SAP Vora, SAP Data Hub, and SAP Data Intelligence in a remarkably short time.

Automation quickly spread across all the development and certification processes: as a result, SAP Data Intelligence itself became a treasure chest full of services and APIs for a complete programmatic control of the product.  In the two previous episodes of Zen and the Art of SAP Data Intelligence, we presented two of these services: the Modeler REST API (Episode 1) and the Connection Manager service (Episode 2).  In this post we will discover the most powerful member of the family: the SAP Data Intelligence System Management Command-Line Client (aka vctl), recently released for productive usage with SAP Data Hub 2.7 .

Prerequisites

• The steps of this post have been performed with Data Hub 2.7.3 and Data Intelligence 1910.1.11, but should be valid for any Data Hub ≥ 2.7 and any Data Intelligence ≥ 1910 versions.
• Recommended browser: Chrome.
• A basic knowledge of Unix commands is required.
• The execution of the snippets proposed in this blog post require a Unix terminal from where the Data Intelligence or Data Hub endpoints are reachable over the network.
• To complete Step 5, the admin credentials of the system tenant are needed, which is usually possible only if you are running an on-prem version of Data Hub or Data Intelligence.

Step 1: where is the System Management Command-Line Client?

This is the most difficult question of the entire post: where is vctl?  If you own an on-prem version of Data Hub or Data Intelligence the answer is a no-brainer: vctl can be downloaded with one click from the System Management application page.

The same executable can be found in the installation package you got from the SAP Market Place.  As you can see from the example below, theinstallation package contains two versions of the client: one for Linux (vctl) and the other for Mac (vctl-darwin).

SAPDataHub-2.7.155-Foundation> ls -1 tools/

cert-driver
checkpoint_admin.sh
checkpoint_config.sh
common.sh
config_helper.py
dh.pem
diagnostics_collector.sh
export.sh
hl-hana-replication.jar
idp_configuration.py
images.sh
import_license.sh
import.sh
json_parser.py
log_collector.py
vctl
vctl-darwin
vsystem.sh

But what if you are using the Data Intelligence service on SCP?  In this case no download option and no installation package, hence no vctl executable.  The reason is clear:  vctl is a tool mostly meant for system administration and dev ops.  Nevertheless, vctl works on the cloud just like it does on prem and I can think of many cases where it can be handy either way.

While waiting for SAP to make vctl available on SCP, cloud users can still try to find a compatible version of the client to install on their local machines.  One possibility might be to setup a trial version by following this excellent tutorial by anisha gupta .  In any case, you can always ask your SAP team for support.

Step 2: my first vctl command

Blame it on my being Italian, but I am firmly convinced that the first thing to do when you approach a new task is to ask for help.  And vctl makes no exception:

./tools/vctl --help

SAP Data Hub System Management CLI

More information at https://help.sap.com/viewer/p/SAP_DATA_HUB

Usage:
  vctl [flags]
  vctl <command>

Available Commands:
  completion Generate completion scripts for bash or zsh
  help       Help about any command
  js         Execute javascript
  login      Log in to SAP Data Hub System Management
  parameter  Manage application parameters (only available to admins)
  passwd     Changes the password of the current user
  policy     Manage policies
  scheduler  Manage application templates and instances
  secret     Manage secrets (only available to admins)
  tenant     Manage tenants and tenant strategies (only available to admins)
  user       Manage users
  version    Display the client and server version of SAP Data Hub System Management
  vrep       Interact with the file repository
  whoami     Print the tenant, username and role associated with the current user.

Flags:
      --conn-file string Path to the connection file (default "/home/d064337/.vsystem/.vsystem_conn")
  -h, --help help for vctl
  -q, --quiet disable logging (overrides --verbose)
  -v, --verbose verbose logging

Use "vctl <command> --help" for more information about a command.

If you get the output above, then you are all set to proceed to the next step.  If not, just drop me a comment and we can check together if it is the case to open an SAP support ticket.

Step 3: login with your DI/DH credentials

Whatever you want to achieve with vctl, it all starts with authentication.  You need the usual four items to successfully compete the login:

1. vsystem ingress URL
2. tenant
3. user
4. password

The login command is pretty straightforward to use:

./tools/vctl login --help

Log in to SAP Data Hub System Management at host and port specified in `<address>` using credentials `<tenant>` `<user>` and `<password>`.

If the -p option is not specified, you will be prompted for the password.

Usage:
vctl login <address> <tenant> <user> [-p <password>] [flags]

Flags:
    --cert stringArray CA certificate for validating the SAP Data Hub System Management peer
-h, --help             help for login
    --insecure         Skip validating SSL connections with installed CA bundles
    --no-proxy         Do not use proxies, even if the proxy environment variables are set
-p, --password string  Password to login with

Global Flags:
    --conn-file string Path to the connection file (default "/home/d064337/.vsystem/.vsystem_conn")
-q, --quiet disable logging (overrides --verbose)
-v, --verbose verbose logging

Better to input your password in the interactive mode instead of using the –password flag:

export cluster_url=<vsystem-ingress-url>
export tenant=<tenant-name>
export user=<user-name>
./tools/vctl login $cluster_url $tenant $user 

Enter password: *************

Step 4: configure a tenant with vctl

You can use vctl to manage and configure pretty much every aspect of your system.  You can create and delete users, change passwords, assign policies, manage your applications, and much more.  As an example, the snippet below shows how to create three new member users in the default tenant.

#! /bin/bash -f

export role="member"
export tenant="default"
export pwd="PwdT0Ch@ng3"
for username in user01 user02 user03
do
    echo creating user $username
    ./tools/vctl user create $tenant $username $pwd $role
done

To delete the users you can use the following:

#! /bin/bash -f

export role="member"
export tenant="default"
export pwd="T0Ch@ng3$$"
for username in user01 user02 user03
do
    echo deleting user $username
    ./tools/vctl user delete $tenant $username
done

Step 5: configure the whole cluster with vctl

Needless to say that the vctl can perform only what the authenticated user is authorized to do.  The two snippets in Step 4, for instance, can be successfully executed only by a tenant administrator.

For on-prem installations, there is a special tenant called system whose administrators have super powers on the whole cluster, especially if they use vctl.  In the example below, the system administrator creates the new tenant called development based on the default strategy strat-default-2.7.155, then create the administrator of this tenant, and finally the member users.

export cluster_url=<vsystem-ingress-url>
export tenant="system"
export user=<system-admin-name>
./tools/vctl --verbose login $cluster_url $tenant $user 

export strategy="strat-default-2.7.155"
export newtenant=<new-tenant-name>
export user=<new-user-name>
export role="tenantAdmin"


echo creating tenant $newtenant
./tools/vctl tenant create $newtenant
./tools/vctl tenant set-strategy $newtenant $strategy

echo creating new administrator user $user for tenant $newtenant
./tools/vctl user create $newtenant $username $pwd $role

export role="member"
export pwd="PwdT0Ch@ng3"
for username in user01 user02 user03
do
    echo creating user $username
    ./tools/vctl user create $newtenant $username $pwd $role
done

Epilog

In this third episode of “Zen and the art of SAP Data Intelligence” we learned how to use the SAP Data Intelligence System Management command-line client.  This tool is the key to automate and scale the system administration and the operations on your Data Intelligence on-prem cluster.  I use it extensively to establish a CI/CD process across my Data Intelligence landscapes.  But this is the topic for another post.

I encourage you to download vctl and try it out with the examples above.  They are just scratching the surface of what this small executable can do.  With a bit of patience and elbow grease you can unlock its full power, and believe me: it is worth the investment.

If you would like to use vctl for something more ambitious than the snippets above and wouldn’t mind a blog post on that, just add your request in the comments.  As usual: all feedbacks are more than welcome.

Thank you for reading!

For the philomaths

Further information about the topics treated in this blog post can be found in the following references:

• The SAP Data Intelligence System Management command-line client is described in the official System Management Command-Line Client Reference.
• For the administration of the on-prem SAP Data Hub systems, you can check the official Administration Guide for SAP Data Hub.
• For the administration of the SAP Data Intelligence cloud services, you can check the official Administration Guide for SAP Data Intelligence.
• The concept of strategies in SAP Data Intelligence and SAP Data Hub is explained here.