SAP Datahub is an unique offering - data orchestration redefined with a flow based design paradigm (Data Pipelines) ,containerized software, and automated fast deployment, scaling, management on kubernetes clusters. Features such as Meta data explorer,Data profiling, catalogue search, Data modeling (Pipelines) lets customers take control of data sets on varied data sources like HDFS, Oracle, DB2,MYSQL,MSSQL,SAP ABAP, SAP Data services, Azure Data Lake (ADL), GCP - Big Query, S3 etc
Currently there are close to 290 predefined Operators to incorporate in a data pipeline they include but are not restricted to : Data Modeling, Data workflow, Kafka, Tensor flow, R, Python,spark submit, HANA client, Write File (HDFS/S3) etc plus support to build customized operators (e.g using own python code) and save then as a docker image, reuse them in several pipelines and scale it massively on Kubernetes clusters.
In this blog I'm going to discuss the underlying technology and implementation aspects of SAP datahub on AWS EKS (Managed K8S in cloud) using some recent experiences from a proof of concept. The installation steps are covered in detail in the SAP guides but this blog is more a real life experience on running SAP datahub on AWS including the steps to expose the datahub UI on a public AWS Route 53 registered domain.
What we need as the base platform to deploy SAP Datahub?
Containers: What is a Container ? Containers are a way of running multiple isolated applications on the same hardware (just like VM's) but it does that using virtual memory on the host and by using the hosts underlying OS for providing basic services to all containers. (VM's on the other had have their own OS), so its basically OS abstraction. As there is no dedicated OS (only virtual memory in play) think of it as a way of running 1000's of services with very less startup time and ability to scale massively.Another important point ,Containers mark a paradigm shift in how applications are packaged and delivered - Development and Operations together, which means that while we are developing our applications we have to be first fundamentally aware of the OS environment, libraries, kernel i.e your *dependencies (* not leaving it for Operations to figure out during deployment) and then develop your application code on top to create an image which when deployed can by itself deploy its dependencies and create a scalable deployment scenario. An example of one such application (among many others in the modern world like Google Docs) is SAP Datahub.
Container Runtime: For running containers on an OS , we need a container runtime which enables containers (e.g. providing a network for Containers to communicate) and providing features such as image management(i.e. the software images which executes in a Container). Good example of a Container Runtime is Docker (Ansible as well from RedHat). When Docker installs on an OS, you would see a network interface docker0 which enables contianer networking, cgroups are used to restricit CPU resources per container. Lets understand from SAP Datahub's Context, each component of SAP datahub like internal HANA DB, Engines like DQP, Flow graph (pipelines), text, graph,consul, distributed log etc run as containers (deployed as a docker image), and one or more such containers running together constitutes a service or functionality, so we also need an environment on top to orchestrate , manage, scale and expose the endpoints of various services in containers running on several nodes - This platform is Kubernetes (Container Orchestration).
Container Registry : Its the place where docker (or the container runtime) stores all images, SAP Stores the relevant docker images in SAP Public repository, when we install datahub on-premise or in cloud AWS/Azure/GCP these images are pulled from SAP Public repositories to the local repositories that the customer sets up. AWS provides Amazon Elastic Container Registry (ECR) for this purpose and has to be setup separately before the SAP datahub install is started. For On-premise customers a local docker registry can be configured( it can be internet facing or otherwise images can be downloaded separately and pushed to the local registry).
Python and Base YAML packages : Some bases packages for Python(python 2.7) and Pyyaml are required as pre-requisites.( check the install guide and the install steps below)
Kubernetes (K8S) : K8S is a container orchestration environment.Think of it as an environment used for managing containers across various nodes. K8s gives rise to the concept of “PODS” - a set of containers working together , having their own network and storage. K8s creates an overlay network (over docker0 we discussed before) to allow PODs to communicate with each other , Kube-DNS helps resolving the names of various services exposed by pods. Google, AWS and Azure all now have managed K8s in cloud.
HELM / TILLER : K8S Package manager, to Deploy packages in K8S. Tiller is the server component and runs as a POD and helm is the client side component(helm command on linux).We will cover some steps to configure it later.
How does this environment benefit us?
Below example is just a food for thought , and maybe not very correct way of solving the problem but just wanted to highlight the benefits.Imagine multiple users in a large Enterprise working on plethora of data that the enterprise generates and the need to get their flavor of data from this ever changing data siloed across several data sources. A data pipeline can work work well to integrate such data and provide real time insights as the data changes, and now imagine 1000's of users in the enterprise running pipeline (inadvertently spinning their own piplelines- containers) , so we have 1000 instances and various containers in action.
Installation
So, Lets Run SAP Datahub on Amazon Web Services EKS , I'm going to focus more on the SAP Datahub installation part on AWS EKS , for more details on how to provision an EKS instance please refer the AWS guide : https://docs.aws.amazon.com/eks/latest/userguide/getting-started.html
Pre-requisites : Get K8S(AWS EKS) up and running (We are going to use 3 EC2 nodes)
Create an IAM role for the EKS Cluster as below :
Create Cluster --> Pre-requisite a VPC should exist:
Add worker nodes once the status of the Cluster is active , add additional EKS nodes :https://docs.aws.amazon.com/eks/latest/userguide/launch-workers.html
Configure Security groups
Please allow inbound communication between EKS nodes on Internal IP's for the K8S network to work.
Checking the EKS K8S and configuring the setup for Datahub deployment:
Login to the master node and use "aws configure" to connect to the EKS user which was created before as the cluster admin :
First login to IAM and generate access key for the EKS admin user:
[ec2-user@ip-~]$ aws configure AWS Access Key ID [****************3NMQ]: AWS Secret Access Key [****************SbIi]: Default region name [ap-northeast-2]: Default output format [json]:
Download and configure Kubectl : https://docs.aws.amazon.com/eks/latest/userguide/install-kubectl.html
Check the Kubernetes enviroment:
[ec2-user@ ~]$ kubectl get nodes NAME STATUS ROLES AGE VERSION ip-<>.ap-northeast-2.compute.internal Ready <none> 23d v1.10.11 ip-<>.ap-northeast-2.compute.internal Ready <none> 24d v1.10.11 ip-<>.ap-northeast-2.compute.internal Ready <none> 24d v1.10.11
Check the existing K8S PODS :
[ec2-user@<> ~]$ kubectl -n kube-system get pods NAME READY STATUS RESTARTS AGE aws-node-292gw 1/1 Running 2 24d aws-node-5d8wt 1/1 Running 2 24d aws-node-whz2n 1/1 Running 1 23d heapster-7ff8d6bf9f-lsk5x 1/1 Running 0 23d kube-dns-5579bfd44-6qpxd 3/3 Running 0 23d kube-proxy-4g6qf 1/1 Running 0 23d kube-proxy-chmgh 1/1 Running 1 24d kube-proxy-sdq7k 1/1 Running 2 24d kubernetes-dashboard-669f9bbd46-ccp8m 1/1 Running 0 23d monitoring-influxdb-cc95575b9-24qxg 1/1 Running 0 23d
Check Kube-DNS is configured correctly , A typical error which means the KUBE DNS is not able to resolve logical names in the K8S network, this can be a problem with they way the K8s was provisioned or with the VPC security groups , this issue can cause problems later when we deploy datahub : https://kubernetes.io/docs/tasks/administer-cluster/dns-debugging-resolution/
root@wg-node1:~# kubectl exec -ti busybox -- nslookup kubernetes.default
Server: 10.100.0.10
Address 1: 10.100.0.10
nslookup: can't resolve 'kubernetes.default'
command terminated with exit code 1
kubectl exec -ti busybox -- nslookup kubernetes.default Server: 10.100.0.10 Address 1: 10.100.0.10 kube-dns.kube-system.svc.cluster.local Name: kubernetes.default Address 1: 10.100.0.1 kubernetes.default.svc.cluster.local
redeploy the cluster if needed or check the connection between nodes works fine.
Download HELM and deploy TILLER:
helm should be available (just type helm on the OS) if not please download the suitable version for your distribution from : https://github.com/helm/helm
[ec2-user@<> linux-amd64]$ kubectl version
Client Version: version.Info{Major:"1", Minor:"10", GitVersion:"v1.10.11", GitCommit:"637c7e288581ee40ab4ca210618a89a555b6e7e9", GitTreeState:"clean", BuildDate:"2018-12-06T02:30:38Z", GoVersion:"go1.9.3", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"10+", GitVersion:"v1.10.11-eks", GitCommit:"6bf27214b7e3e1e47dce27dcbd73ee1b27adadd0", GitTreeState:"clean", BuildDate:"2018-12-04T13:33:10Z", GoVersion:"go1.9.3", Compiler:"gc", Platform:"linux/amd64"}
download Helm Archive on OS and Extract:
[ec2-user@<> ~]$ ls -lrt|grep helm
-rw-rw-r-- 1 ec2-user ec2-user 30054400 Jan 29 06:50 helm-v2.9.1-linux-amd64.tar
[ec2-user@<> ~]$ cd linux-amd64
[ec2-user@<> linux-amd64]$ ls -lrt
total 29348
-rwxr-xr-x 1 ec2-user ec2-user 30033696 May 14 2018 helm
-rw-r--r-- 1 ec2-user ec2-user 3310 May 14 2018 README.md
-rw-r--r-- 1 ec2-user ec2-user 11373 May 14 2018 LICENSE
copy it to the bin locations under your $PATH env:
[ec2-user@<> linux-amd64]$ which helm
/usr/local/bin/helm
Create Cluster Role binding and start Tiller (Runs as a POD, we should be able to see it running later)
$kubectl create serviceaccount --namespace kube-system tiller $ kubectl create clusterrolebinding tiller --clusterrole=cluster-admin --serviceaccount=kube-system:default $ helm init --service-account tiller ##additional Step : I had to create this additional clusterrole binding as the dtaahub install later gave an error root@ip-10-0-1-82:~# kubectl create clusterrolebinding tiller-clusterrule --clusterrole=cluster-admin --serviceaccount=kube-system:tiller clusterrolebinding.rbac.authorization.k8s.io "tiller-clusterrule" created
[ec2-user@<> linux-amd64]$ kubectl -n kube-system get pods NAME READY STATUS RESTARTS AGE aws-node-292gw 1/1 Running 2 24d aws-node-5d8wt 1/1 Running 2 24d aws-node-whz2n 1/1 Running 1 23d heapster-7ff8d6bf9f-lsk5x 1/1 Running 0 23d kube-dns-5579bfd44-6qpxd 3/3 Running 0 23d kube-proxy-4g6qf 1/1 Running 0 23d kube-proxy-chmgh 1/1 Running 1 24d kube-proxy-sdq7k 1/1 Running 2 24d kubernetes-dashboard-669f9bbd46-ccp8m 1/1 Running 0 23d monitoring-influxdb-cc95575b9-24qxg 1/1 Running 0 23d prodding-walrus-kube-lego-85467b8dd4-vqxm7 1/1 Running 0 12d tiller-deploy-f9b8476d-wnmgj 1/1 Running 0 23d
Configure AWS ECS : Elastic Container Registry : Remember we discussed in our description of datahub above that we need a Container registry to store various images of software, during the SAP datahub install we would be pulling software images form SAP Public repository to the ECS registry.
[ec2-user@ip-<> linux-amd64]$ aws ecr get-login --no-include-email docker login -u AWS -p eyJwYXlsb2FkIjoiMlg4eXFmT050N3JlN29ReUovenNWWWxqU0d3ZkFEeElkYXRlZFBMMlJKWk9RNjE3WTRiMmRJOUVYMVRuV0tRUk9CcjNMN3B0S2tiU0syQi9iMGF2Fqa200emNiNWVNMDFWaEdpNVdoTGNncG9BbkdCd2FZRjVEdFFPM2M1Ry9qeEJVTzJ3ZE5xV1ZaQ24vRGlOSyttMVlaV04xaUJoODZncmFPUGlCcmU1SkRlMUc4T25MNUJFQ0txZ . ... . . . . . . . . ##Run the complete putput of the above command : [ec2-user@<> linux-amd64]$ docker login -u AWS -p eyJwYXlsb2FkIjoiMlg4eXFmT050N3JlN29ReUovenNWWWxqU0d3ZkFEeElkYXRlZFBMMlJKWk9RNjE3WTRiMmRJOUVYMVRuV0tRUk9CcjNMN................ https://<>.dkr.ecr.ap-northeast-2.amazonaws.com WARNING! Using --password via the CLI is insecure. Use --password-stdin. WARNING! Your password will be stored unencrypted in /home/ec2-user/.docker/config.json. Configure a credential helper to remove this warning. See https://docs.docker.com/engine/reference/commandline/login/#credentials-store Login Succeeded Create below repositories : aws ecr create-repository --repository-name=com.sap.datahub.linuxx86_64/vora-dqp aws ecr create-repository --repository-name=com.sap.datahub.linuxx86_64/vora-dqp-textanalysis aws ecr create-repository --repository-name=com.sap.datahub.linuxx86_64/spark-datasourcedist aws ecr create-repository --repository-name=com.sap.hana.container/base-opensuse42.3-amd64 aws ecr create-repository --repository-name=com.sap.datahub.linuxx86_64/vora-deployment-operator aws ecr create-repository --repository-name=com.sap.datahub.linuxx86_64/security-operator aws ecr create-repository --repository-name=com.sap.datahub.linuxx86_64/init-security aws ecr create-repository --repository-name=com.sap.datahub.linuxx86_64/uaa aws ecr create-repository --repository-name=com.sap.datahub.linuxx86_64/opensuse-leap aws ecr create-repository --repository-name=com.sap.datahub.linuxx86_64/vsystem-vrep aws ecr create-repository --repository-name=com.sap.datahub.linuxx86_64/vsystem aws ecr create-repository --repository-name=com.sap.datahub.linuxx86_64/vsystem-auth aws ecr create-repository --repository-name=com.sap.datahub.linuxx86_64/vsystem-teardown aws ecr create-repository --repository-name=com.sap.datahub.linuxx86_64/vsystem-module-loader aws ecr create-repository --repository-name=com.sap.datahub.linuxx86_64/app-base aws ecr create-repository --repository-name=com.sap.datahub.linuxx86_64/flowagent aws ecr create-repository --repository-name=com.sap.datahub.linuxx86_64/app-base aws ecr create-repository --repository-name=com.sap.datahub.linuxx86_64/vora-license-manager aws ecr create-repository --repository-name=com.sap.datahub.linuxx86_64/vsystem-shared-ui aws ecr create-repository --repository-name=com.sap.datahub.linuxx86_64/vsystem-ui aws ecr create-repository --repository-name=com.sap.datahub.linuxx86_64/vsystem-voraadapter aws ecr create-repository --repository-name=elasticsearch/elasticsearch-oss aws ecr create-repository --repository-name=fabric8/fluentd-kubernetes aws ecr create-repository --repository-name=grafana/grafana aws ecr create-repository --repository-name=kibana/kibana-oss aws ecr create-repository --repository-name=google_containers/kube-state-metrics aws ecr create-repository --repository-name=nginx aws ecr create-repository --repository-name=prom/alertmanager aws ecr create-repository --repository-name=prom/node-exporter aws ecr create-repository --repository-name=prom/prometheus aws ecr create-repository --repository-name=prom/pushgateway aws ecr create-repository --repository-name=consul aws ecr create-repository --repository-name=nats-streaming aws ecr create-repository --repository-name=vora/hello-world
Goto to the ECS page from AWS management console :
As we run through the SAP datahub install steps we would see these repositories populating with the images.
Setup Environment variables for NAMESPACE and Container Registry:
check the URI of the ECS registry from the above step and configure the below environment variables. NAMESPACE is the namespace under which all our SAP datahub PODS/containers will be deployed. We would be able to visualize this in a bettwe way with K8S Dadhboard.
[ec2-user@<> linux-amd64]$ export NAMESPACE=datahub24 [ec2-user@<> linux-amd64]$ export DOCKER_REGISTRY=3*******.dkr.ecr.ap-northeast-2.amazonaws.com
DEPLOY K8S Dashoard and access it by opening a tunnel from local laptop, its quite important to have it running before deploying datahub as its very useful for troubleshooting.
K8S dashbaord deployed using AWS documentation https://docs.aws.amazon.com/eks/latest/userguide/dashboard-tutorial.html
Here is how to run it :
[ec2-user@<> linux-amd64]$ kubectl proxy Starting to serve on 127.0.0.1:8001
on your laptop open a tunnel :
Now access the dashboard : http://localhost:8001/api/v1/namespaces/kube-system/services/https:kubernetes-dashboard:/proxy/#!/lo...
Generate a Token to login:
[ec2-user@<> ~]$ kubectl -n kube-system describe secret $(kubectl -n kube-system get secret | grep eks-admin | awk '{print $1}')
Name: eks-admin-token-cmljn
Namespace: kube-system
Labels: <none>
Annotations: kubernetes.io/service-account.name=eks-admin
kubernetes.io/service-account.uid=dca7931a-23a6-11e9-808f-0ae755ba91d4
Type: kubernetes.io/service-account-token
Data
====
token: <use the token from here>
ca.crt: 1025 bytes
namespace: 11 bytes
Create a Storage class : If the EKS cluster was provisioned without a storage class, add a storage class as per : https://docs.aws.amazon.com/eks/latest/userguide/storage-classes.html
Make sure yo uhave a default annotation if gp2 storage for example is the default storagre
[ec2-user@<> ~]$ kubectl get storageclass NAME PROVISIONER AGE eks-cluster-sc kubernetes.io/aws-ebs 24d gp2 (default) kubernetes.io/aws-ebs 23d vrep-datahub24 sap.com/vrep 23d
vrep-datahub24 : this storage class was created automatically during the install and its basically used for different piplelines
SAP DATAHUB Installation : I'm using he manual install procedure (alternate install procedure)
Download the SAP datahub software archive from SAP market place for Datahub 2.4: DHFOUNDATION04_0-80004015.ZIP
[ec2-user@<> ~]$ ls -lrt|grep SAPData drwxrwxr-x 8 ec2-user ec2-user 189 Jan 30 04:43 SAPDataHub-2.4.63-Foundation [ec2-user@<> ~]$ cd SAPDataHub-2.4.63-Foundation [ec2-user@<> SAPDataHub-2.4.63-Foundation]$ ls -lrt total 128 -rw-r--r-- 1 ec2-user ec2-user 1717 Dec 3 19:05 license_agreement -rwxr-xr-x 1 ec2-user ec2-user 116588 Dec 3 19:05 install.sh -rw-r--r-- 1 ec2-user ec2-user 2953 Dec 3 19:05 dev-config.sh.tpl drwxrwxr-x 6 ec2-user ec2-user 93 Jan 29 09:09 slplugin drwxrwxr-x 7 ec2-user ec2-user 106 Jan 29 09:09 validation drwxrwxr-x 2 ec2-user ec2-user 279 Jan 29 09:09 tools drwxrwxr-x 9 ec2-user ec2-user 167 Jan 30 04:19 logs drwxrwxr-x 6 ec2-user ec2-user 74 Jan 30 04:25 deployment
I had no special conditions to pass to the install prompt but depending on the case please refer the instillation guide , a single Install command does it all, i was prompted to enter my SAP OSS credentials to authorize the download of the Software.
[root@<> SAPDataHub-2.4.63-Foundation]# ./install.sh 2019-01-30T04:19:41+0000 [INFO] Running pre-flight checks 2019-01-30T04:19:41+0000 [INFO] Checking if python2.7 is installed...OK! 2019-01-30T04:19:41+0000 [INFO] Checking if pyyaml is installed...OK! 2019-01-30T04:19:41+0000 [INFO] Checking kubectl configuration...OK! 2019-01-30T04:19:41+0000 [INFO] Checking kubernetes client is 1.9.x or 1.10.x or 1.11.x...OK! 2019-01-30T04:19:41+0000 [INFO] Checking kubernetes version is 1.9.x or 1.10.x or 1.11.x...OK! 2019-01-30T04:19:41+0000 [INFO] Checking whether namespace exists or possible to create...OK! 2019-01-30T04:19:42+0000 [INFO] Trying to access SAP Container Artifactory by pulling an image... Error response from daemon: Get https://73554900100900002861.dockersrv.repositories.sap.ondemand.com/v2/com.sap.datahub.linuxx86_64/hello-sap/manifests/1.1: no basic auth credentials You need to login to access SAP Container Artifactory. 1) Login with Technical User 2) Login with S-User Please select one of the login options: 2 Please provide username: I338450 Please provide password: WARNING! Your password will be stored unencrypted in /root/.docker/config.json. Configure a credential helper to remove this warning. See https://docs.docker.com/engine/reference/commandline/login/#credentials-store Login Succeeded 2019-01-30T04:20:01+0000 [INFO] Checking docker.io access...OK! 2019-01-30T04:20:06+0000 [INFO] Checking push to internal docker registry...OK! 2019-01-30T04:20:06+0000 [INFO] Checking pull from internal docker registry...OK! 2019-01-30T04:20:07+0000 [INFO] Checking helm client is 2.8.x, 2.9.x, 2.10.x or 2.11.x...OK! 2019-01-30T04:20:07+0000 [INFO] Checking if helm installed and initialized...OK! 2019-01-30T04:20:10+0000 [INFO] Wait until tiller is ready... 2019-01-30T04:20:10+0000 [INFO] Checking if helm is ready...OK! 2019-01-30T04:20:10+0000 [INFO] Checking if there is no failed helm chart in the namespace...OK! 2019-01-30T04:20:10+0000 [INFO] Checking if there are any existing persistent volume claims...OK! 2019-01-30T04:20:11+0000 [INFO] End of pre-flight checks
The install will progress from here on and download the images in the ECR address updated using the environment variables: Inputs needed
- Accept and Authorize download and installation of docker images.
- Provide Subject alternate names for the SSL certificates < this is needed if you want to utilize K8S generated certificates, in our case we wont be using them>
- Provide system user Password and vora default tenant password
By running the SAP Data Hub installer and by using built-in operators of SAP Data Hub, Docker images will be built by automatically downloading and installing: (a) Docker images from SAP Docker Registry; (b) Docker images from third party registries; and (c) open source prerequisites from third party open source repositories. The Docker images from the SAP Docker Registry are part of the SAP Data Hub product. Use of these images is governed by the terms of your commercial agreement with SAP for the SAP Data Hub. The Docker images from third party registries and open source prerequisites from third party open source repositories (collectively, the “Third Party Prerequisites”) are prerequisites of SAP Data Hub that usually would have to be downloaded and installed by customers from third party repositories before deploying the SAP Data Hub. For the customers' convenience, the SAP Data Hub installer and built-in operators automatically download and install the Third Party Prerequisites on behalf of the customer. The Third Party Prerequisites are NOT part of the SAP Data Hub and SAP does not accept any responsibility for the Third Party Prerequisites, including, providing support. Use of the Third Party Prerequisites is solely at customers’ risk and subject to any third party licenses applicable to the use of such prerequisites. Customers are responsible for keeping the Third Party Prerequisites up-to-date, and are asked to make use of the respective community support and / or to consider commercial support offerings. The Third Party Prerequisites and associated license information are listed in the Release Note for SAP Data Hub that is published at the download site for SAP Data Hub. I authorize the download and installation of Docker images from the SAP Docker Registry and Third Party Prerequisites from third party repositories, and acknowledge the foregoing disclaimer. (yes/no) yes .... ...... ....... No SSL certificate has been provided via the --provide-certs parameter. The SAP Data Hub installer will generate a self-signed certificate for TLS and JWT. Please enter the SAN (Subject Alternative Name) for the certificate, which must match the fully qualified domain name (FQDN) of the Kubernetes node to be accessed externally: ap-northeast-2.compute.amazonaws.com SAP Data Hub System Tenant Administrator Credentials Provide a password for the "system" user of "system" tenant. The password must have 8-255 characters and must contain lower case, upper case, numerical and special characters . @ # $ %% * - + _ ? ! Please enter a password for "system" user of "system" tenant: Please reenter your password: SAP Dat Hub Initial Tenant Administrator Credentials Provide a username and password for administrator user of "default" tenant. The username must have at least 4 and at most 60 characters Allowed characters: alphabetic(only lowercase), digits and hyphens Username is not allowed to begin/end with hyphens and cannot contain multiple consecutive hyphens Please enter a username for default tenant: vora Do you want to use the same "system" user password for "vora" user of "default" tenant? (yes/no) yes Do you want to configure security contexts for Hadoop/Kerberized Hadoop? (yes/no) no 2019-01-30T04:24:32+0000 [INFO] Configuring contexts with: python2.7 configure_contexts.py -a -n --set Vora_JWT_Issuer_NI.default --set Vora_Default_TLS_Configuration_NI.default secret "vora.conf.secop.contexts" created secret "vora.conf.secop.contexts" labeled 2019-01-30T04:24:33+0000 [INFO] Vora streaming tables require Vora's checkpoint store\n Enable Vora checkpoint store? (yes/no) no ###### Configuration Summary ####### installer: AUDITLOG_MODE: production CERT_DOMAIN: ap-northeast-2.compute.amazonaws.com CHECKPOINT_STORE_TYPE: '' CHECKPOINT_STORE_TYPE_RAW: '' CLUSTER_HTTPS_PROXY: '' CLUSTER_HTTP_PROXY: '' CLUSTER_NO_PROXY: '' CONSUL_STORAGE_CLASS: '' CUSTOM_DOCKER_LOG_PATH: '' DIAGNOSTIC_STORAGE_CLASS: '' DISABLE_INSTALLER_LOGGING: '' DISK_STORAGE_CLASS: '' DLOG_STORAGE_CLASS: '' DOCKER_REGISTRY: 361176622288.dkr.ecr.ap-northeast-2.amazonaws.com ENABLE_CHECKPOINT_STORE: 'false' ENABLE_DIAGNOSTIC_PERSISTENCY: 'no' ENABLE_NETWORK_POLICIES: 'no' ENABLE_RBAC: 'yes' ENABLE_UAA: 'true' HANA_STORAGE_CLASS: '' IMAGE_PULL_SECRET: '' PACKAGE_VERSION: 2.4.63 PV_STORAGE_CLASS: '' TILLER_NAMESPACE: '' USE_K8S_DISCOVERY: 'yes' VALIDATE_CHECKPOINT_STORE: '' VFLOW_IMAGE_PULL_SECRET: '' VFLOW_REGISTRY: '' VORA_ADMIN_USERNAME: vora VORA_FLAVOR: '' VORA_VSYSTEM_DEFAULT_TENANT_NAME: default VSYSTEM_LOAD_NFS_MODULES: 'yes' VSYSTEM_STORAGE_CLASS: '' ###################################### Do you want to start the installation with this configuration (yes/no) yes 2019-01-30T04:24:43+0000 [INFO] Updating installer configuration... secret "installer-config" created secret "installer-config" labeled OK! 2019-01-30T04:24:44+0000 [INFO] Image already exists on the target registry "361176622288.dkr.ecr.ap-northeast-2.amazonaws.com" as "com.sap.hana.container/base-opensuse42.3-amd64:2.03.031.00-3.1.0", will not mirror 2019-01-30T04:24:44+0000 [INFO] Image already exists on the target registry "361176622288.dkr.ecr.ap-northeast-2.amazonaws.com" as "consul:0.9.0", will not mirror 2019-01-30T04:24:45+0000 [INFO] Image already exists on the target registry "361176622288.dkr.ecr.ap-northeast-2.amazonaws.com" as "com.sap.datahub.linuxx86_64/vora-dqp:2.4.46", will not mirror 2019-01-30T04:24:45+0000 [INFO] Image already exists on the target registry "361176622288.dkr.ecr.ap-northeast-2.amazonaws.com" as "com.sap.datahub.linuxx86_64/vora-deployment-operator:2.4.46", will not mirror ..... .. ..............
Note in the screenshot above, I had already run the install once before and hence the docker images were already downloaded.
Keep monitoring the PODS spinning up in K8S dashboard and check for example if the Persitent volume claims on the storage volumes are going through well or if there are certain PODS failing with errors , if such is the case, click on the LOGS to see the error messages on each such failing pods. Example below , see the HANA POD:
Launch a shell inside a POD --> click on the POD --> from right hand top select "EXEC" :
At the End of installation , a validation check is done on Vora-cluster, sparkonk8s etc.
2019-01-30T04:36:03+0000 [INFO] Labelling persistent volume claims... persistentvolumeclaim "data-log-hana-0" labeled persistentvolumeclaim "data-vora-disk-0" labeled persistentvolumeclaim "data-vora-dlog-0" labeled persistentvolumeclaim "data-vora-dlog-1" labeled persistentvolumeclaim "data-vora-dlog-2" labeled persistentvolumeclaim "datadir-vora-consul-0" labeled persistentvolumeclaim "datadir-vora-consul-1" labeled persistentvolumeclaim "datadir-vora-consul-2" labeled persistentvolumeclaim "trace-hana-0" labeled persistentvolumeclaim "layers-volume-vsystem-vrep-0" labeled 2019-01-30T04:36:05+0000 [INFO] Validating... 2019-01-30T04:36:05+0000 [INFO] Running validation for vora-cluster...OK! 2019-01-30T04:37:59+0000 [INFO] Running validation for vora-sparkonk8s...OK! 2019-01-30T04:38:41+0000 [INFO] Running validation for vora-vsystem...OK! 2019-01-30T04:38:48+0000 [INFO] Running validation for vora-diagnostic...OK! 2019-01-30T04:43:54+0000 [INFO] Running validation for datahub-app-base-db...OK!
Once the install finishes Verify the various PODS are running under the datahub24 namespace:
[ec2-user@<> ~]$ kubectl get pods --all-namespaces
NAMESPACE NAME READY STATUS RESTARTS AGE
datahub24 audit-log-viewer-5sd29-97d8b6dc7-tjkg4 3/3 Running 0 2d
datahub24 audit-log-viewer-pkqjv-85bdcf9867-s7vwm 3/3 Running 0 2d
datahub24 auditlog-787bfc46f8-w8j9j 1/1 Running 0 23d
datahub24 connection-management-2klfm-b5d8b9d4c-8gcjg 3/3 Running 0 12d
datahub24 connection-management-mtknw-5bc4cf5c84-ftjpx 3/3 Running 0 22d
datahub24 connection-management-xws79-5b44778855-hqb6m 3/3 Running 0 12d
datahub24 datahub-app-db-f2xp5-5bd9dd64bb-6ll8f 3/3 Running 0 23d
datahub24 datahub-app-db-kl85c-7dc757c7d5-n4cb4 3/3 Running 0 23d
datahub24 diagnostics-elasticsearch-0 0/1 CrashLoopBackOff 6390 23d
datahub24 diagnostics-elasticsearch-retention-policy-6498488546-8st5n 1/1 Running 0 23d
datahub24 diagnostics-fluentd-76jvh 1/1 Running 0 6d
datahub24 diagnostics-fluentd-bh7zx 1/1 Running 0 23d
datahub24 diagnostics-fluentd-qdr77 1/1 Running 0 23d
datahub24 diagnostics-grafana-6cc75dfdd-gh5hp 2/2 Running 0 23d
datahub24 diagnostics-kibana-6d789c9789-nd5bj 2/2 Running 0 23d
datahub24 diagnostics-prometheus-kube-state-metrics-6c4b796879-kbkbg 1/1 Running 0 23d
datahub24 diagnostics-prometheus-node-exporter-dqhxb 1/1 Running 0 23d
datahub24 diagnostics-prometheus-node-exporter-hp8n6 1/1 Running 0 23d
datahub24 diagnostics-prometheus-node-exporter-lqns7 1/1 Running 0 23d
0 6d
datahub24 diagnostics-prometheus-pushgateway-58bb88c4d6-c6qxl 2/2 Running 0 6d
0 6d
datahub24 diagnostics-prometheus-server-0 1/1 Running 0 2d
datahub24 flowagent-5cs77-77b848f7b6-tl9lh 3/3 Running 0 2d
datahub24 flowagent-nzp85-cd7855484-p762j 3/3 Running 0 2d
datahub24 hana-0 1/1 Running 0 23d
datahub24 internal-comm-secret-gen-8wgpd 0/1 Completed 0 23d
datahub24 launchpad-ddppk-55c669755c-2rgzt 3/3 Running 0 12d
datahub24 launchpad-h9gqt-7cdb6dbcd8-fdpqt 3/3 Running 0 12d
datahub24 launchpad-hpvfp-64fc44dc89-9n465 3/3 Running 0 22d
datahub24 launchpad-js56k-865f9bbff8-wlbvk 3/3 Running 0 12d
datahub24 launchpad-jxhsq-787d74cfd4-v8ccs 3/3 Running 0 12d
datahub24 license-management-bdtq6-85bd9b88b7-k88rc 3/3 Running 0 23d
datahub24 license-management-nbpt9-64bbc799c-zkl87 3/3 Running 0 22d
datahub24 metadata-explorer-mnxlg-69768799f-kjkvf 3/3 Running 1 12d
datahub24 metadata-explorer-pq6mr-5ddb5dd8d8-x2wmh 3/3 Running 0 22d
datahub24 metadata-explorer-rrf5s-84f9686975-qnvzc 3/3 Running 1 12d
datahub24 metadata-explorer-z995p-64c485856c-kk8jd 3/3 Running 0 8d
datahub24 monitoring-29qx9-8544fb8594-d248s 3/3 Running 0 10d
datahub24 monitoring-6tccm-65cdc5555c-t5hsw 3/3 Running 0 2d
datahub24 monitoring-pm4hr-84c749f85b-bs9qk 3/3 Running 2 12d
datahub24 monitoring-r5pnh-9856594fc-btbk7 3/3 Running 0 22d
datahub24 scheduler-glwjq-676bb758fb-ljpd4 3/3 Running 0 22d
datahub24 shared-2nj2x-679666c7fd-mnr6q 3/3 Running 0 22d
datahub24 system-management-8p7fp-6bb84c4cd-scbsq 3/3 Running 0 12d
datahub24 system-management-9kw5g-6787955c57-mktlw 3/3 Running 0 12d
datahub24 system-management-g2k7d-9f8894996-lxnsb 3/3 Running 0 22d
datahub24 system-management-lm29n-65d6cdd67c-jv8k6 3/3 Running 0 12d
datahub24 system-management-pjsts-754dff5c59-9lhl2 3/3 Running 0 12d
datahub24 task-application-6rgnp-5d6fb96d8f-fszgz 3/3 Running 0 2d
datahub24 uaa-5db6c9d965-kvpzf 2/2 Running 1 9d
datahub24 vora-catalog-54b6c76cb9-flw5b 2/2 Running 2 23d
datahub24 vora-config-init-jrwh5 0/2 Completed 0 23d
datahub24 vora-consul-0 1/1 Running 0 23d
datahub24 vora-consul-1 1/1 Running 0 23d
datahub24 vora-consul-2 1/1 Running 0 23d
datahub24 vora-deployment-operator-c8c848767-ghklm 1/1 Running 0 23d
datahub24 vora-disk-0 2/2 Running 0 6d
datahub24 vora-dlog-0 2/2 Running 0 23d
datahub24 vora-dlog-1 2/2 Running 0 23d
datahub24 vora-dlog-2 2/2 Running 0 23d
datahub24 vora-dlog-admin-t7znm 0/2 Completed 0 23d
datahub24 vora-doc-store-755689c9d4-k77b6 2/2 Running 2 23d
datahub24 vora-graph-7bbdcbff78-8g4hd 2/2 Running 2 23d
datahub24 vora-landscape-59664c66fc-cqbfd 2/2 Running 1 23d
datahub24 vora-nats-streaming-9d44586f8-2ckqr 1/1 Running 0 23d
datahub24 vora-relational-fd7fb7d9-qnqmz 2/2 Running 2 23d
datahub24 vora-security-operator-946cbf44f-bw8q9 1/1 Running 0 23d
datahub24 vora-spark-resource-staging-server-7548d94c9b-kkkgn 1/1 Running 0 23d
datahub24 vora-textanalysis-6fc6db7f9b-7qb27 1/1 Running 0 23d
datahub24 vora-time-series-98c95d7-8m9gl 2/2 Running 1 23d
datahub24 vora-tools-225cq-7dc4c9cfb5-sq5s6 3/3 Running 0 10d
datahub24 vora-tools-kmvwr-559dbb64c4-b29xw 3/3 Running 0 22d
datahub24 vora-tools-rhkb9-575dbdfc85-brv52 3/3 Running 0 12d
datahub24 vora-tx-broker-6864547954-6lm2d 2/2 Running 2 23d
datahub24 vora-tx-coordinator-6544b9bfd6-pcsjg 2/2 Running 1 23d
datahub24 vora-tx-lock-manager-5c8699cb6-mhmk9 2/2 Running 2 23d
datahub24 voraadapter-gq4dt-669f4b4944-r54f8 3/3 Running 0 23d
datahub24 voraadapter-pdkgm-56fb9f466d-kh5lv 3/3 Running 0 23d
datahub24 vsystem-fdcf96cbf-mbhn8 2/2 Running 0 23d
datahub24 vsystem-module-loader-bbprd 1/1 Running 0 23d
datahub24 vsystem-module-loader-c57x6 1/1 Running 0 23d
datahub24 vsystem-module-loader-v5fqt 1/1 Running 0 23d
datahub24 vsystem-vrep-0 1/1 Running 0 23d
default busybox 1/1 Running 575 23d
ingress-nginx nginx-ingress-controller-75cd585bbc-sjxf4 1/1 Running 0 13d
kube-system aws-node-292gw 1/1 Running 2 25d
kube-system aws-node-5d8wt 1/1 Running 2 25d
kube-system aws-node-whz2n 1/1 Running 1 23d
kube-system heapster-7ff8d6bf9f-lsk5x 1/1 Running 0 23d
kube-system kube-dns-5579bfd44-6qpxd 3/3 Running 0 23d
kube-system kube-proxy-4g6qf 1/1 Running 0 23d
kube-system kube-proxy-chmgh 1/1 Running 1 25d
kube-system kube-proxy-sdq7k 1/1 Running 2 25d
kube-system kubernetes-dashboard-669f9bbd46-ccp8m 1/1 Running 0 23d
kube-system monitoring-influxdb-cc95575b9-24qxg 1/1 Running 0 23d
kube-system prodding-walrus-kube-lego-85467b8dd4-vqxm7 1/1 Running 0 12d
kube-system tiller-deploy-f9b8476d-wnmgj 1/1 Running 0 23d
Access Datahub application : Datahub application is exposed by vsystem Service, Open K8s dashboard and select your namespace(datahub24 in my case) goto "Discovery and Load Balancing" --> services --> find vsystem and the port -->
Open a ssh tunnel from your laptop to this port 31855 in the above case:
C:\Users\i338450>ssh -L 31855:localhost:31855 -i EMT_KEY.pem ec2-user@13.125.144.154
Access the application on : https://localhost:31855
EXPOSING Datahub App Endpoint on AWS Route 53 Domain
I had registered Domain on AWS : datahubpoc.com beforehand, so wanted to expose the Datahub vsystem App endpoint as https://poc1.<..>.com. So first create a Domain using route 53, once this is done we would be able to add the CNAME = poc1.<...>.com pointing to the Ingress IP address (Follow the steps below):
As per the installation guide, we first generate an open ssl certificate and expose it as a secret in K8S :
[root@<> ec2-user]# openssl req -x509 -nodes -days 365 -newkey rsa:2048 -keyout /tmp/tls.key -out /tmp/tls.crt -subj "/CN=poc1.datahubpoc.com"
Generating a 2048 bit RSA private key
..+++
.........................................................................................+++
writing new private key to '/tmp/tls.key'
-----
[root@<>-user]# kubectl -n datahub24 create secret tls vsystem-tls-certs --key /tmp/tls.key --cert /tmp/tls.crt
secret "vsystem-tls-certs" created
[root@<> ec2-user]# kubectl apply -f https://raw.githubusercontent.com/kubernetes/ingress-nginx/master/deploy/mandatory.yaml
kubectl apply -f https://raw.githubusercontent.com/kubernetes/ingress-nginx/master/deploy/provider/aws/service-l4.yaml
kubectl apply -f https://raw.githubusercontent.com/kubernetes/ingress-nginx/master/deploy/provider/aws/patch-configmap-l4.yaml
namespace "ingress-nginx" created
configmap "nginx-configuration" created
configmap "tcp-services" created
configmap "udp-services" created
serviceaccount "nginx-ingress-serviceaccount" created
clusterrole.rbac.authorization.k8s.io "nginx-ingress-clusterrole" created
role.rbac.authorization.k8s.io "nginx-ingress-role" created
rolebinding.rbac.authorization.k8s.io "nginx-ingress-role-nisa-binding" created
clusterrolebinding.rbac.authorization.k8s.io "nginx-ingress-clusterrole-nisa-binding" created
deployment.apps "nginx-ingress-controller" created
We would eventually need TLS/SSL on our endpoint https://poc1.<..>.com , this can be easily done with Free Certificate authority LetsEncrypt : more details : https://docs.bitnami.com/kubernetes/how-to/secure-kubernetes-services-with-ingress-tls-letsencrypt/
For https we have to install kube-lego chart , email address is mandatory, kube-lego will pick up the change to the Ingress object, request the certificate from LetsEncrypt and store it in the “vsystem-tls-cert” Secret.(which we would be creating subsequently) In turn, the NGINX Ingress Controller will read the TLS configuration and load the certificate from the Secret. Once the NGINX server is updated, a visit to the domain in the browser should present the datahub App site over a secure TLS connection.
[ec2-user@<> tmp]$ helm install stable/kube-lego --namespace kube-system --set config.LEGO_EMAIL=<Enter Your Email>,config.LEGO_URL=https://acme-v01.api.letsencrypt.org/directory
NAME: prodding-walrus
LAST DEPLOYED: Sat Feb 9 20:29:47 2019
NAMESPACE: kube-system
STATUS: DEPLOYED
RESOURCES:
==> v1beta1/Deployment
NAME DESIRED CURRENT UP-TO-DATE AVAILABLE AGE
prodding-walrus-kube-lego 1 0 0 0 0s
==> v1/Pod(related)
NAME READY STATUS RESTARTS AGE
prodding-walrus-kube-lego-85467b8dd4-vqxm7 0/1 Pending 0 0s
NOTES:
This chart installs kube-lego to generate TLS certs for Ingresses.
NOTE:
> kube-lego is in maintenance mode only. There is no plan to support any new
> features. The officially endorsed successor is cert-manager
> https://hub.kubeapps.com/charts/stable/cert-manager
EXAMPLE INGRESS YAML:
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
name: example
namespace: foo
annotations:
kubernetes.io/ingress.class: nginx
# Add to generate certificates for this ingress
kubernetes.io/tls-acme: 'true'
spec:
rules:
- host: www.example.com
http:
paths:
- backend:
serviceName: exampleService
servicePort: 80
path: /
tls:
# With this configuration kube-lego will generate a secret in namespace foo called `example-tls`
# for the URL `www.example.com`
- hosts:
- "www.example.com"
secretName: example-tls
We are now ready to deploy the Ingress:
My ingress YAML File:
[ec2-user@<> ~]$ cat ingress.yaml
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
annotations:
ingress.kubernetes.io/force-ssl-redirect: "true"
ingress.kubernetes.io/proxy-body-size: 500m
ingress.kubernetes.io/secure-backends: "true"
kubernetes.io/ingress.class: nginx
kubernetes.io/tls-acme: "true"
nginx.ingress.kubernetes.io/backend-protocol: HTTPS
nginx.ingress.kubernetes.io/proxy-body-size: 500m
nginx.ingress.kubernetes.io/proxy-buffer-size: 16k
nginx.ingress.kubernetes.io/proxy-connect-timeout: "30"
nginx.ingress.kubernetes.io/proxy-read-timeout: "1800"
nginx.ingress.kubernetes.io/proxy-send-timeout: "1800"
nginx.ingress.kubernetes.io/ssl-passthrough: "true"
name: vsystem
spec:
rules:
- host: <DNS_REPLACE>
http:
paths:
- backend:
serviceName: vsystem
servicePort: 8797
path: /
tls:
- hosts:
- <DNS_REPLACE>
secretName: vsystem-tls-certs
Note Carefully replace <DNS_REPLACE> as below :
[ec2-user@<> ~]$ export dns_domain=poc1.<....>.com
[ec2-user@<> ~]$ cat ingress.yaml| sed "s/<DNS_REPLACE>/${dns_domain}/g" | kubectl -n datahub24 apply -f -
ingress.extensions "vsystem" created
describe your Ingress :
[ec2-user@<> ~]$ kubectl -n datahub24 describe ingress
Name: vsystem
Namespace: datahub24
Address: adb19<.......>.ap-northeast-2.elb.amazonaws.com
Default backend: default-http-backend:80 (<none>)
TLS:
vsystem-tls-certs terminates poc1.datahubpoc.com
Rules:
Host Path Backends
---- ---- --------
poc1.datahubpoc.com
/ vsystem:8797 (<none>)
Annotations:
ingress.kubernetes.io/proxy-body-size: 500m
nginx.ingress.kubernetes.io/backend-protocol: HTTPS
nginx.ingress.kubernetes.io/proxy-connect-timeout: 30
nginx.ingress.kubernetes.io/ssl-passthrough: true
nginx.ingress.kubernetes.io/proxy-buffer-size: 16k
nginx.ingress.kubernetes.io/proxy-read-timeout: 1800
ingress.kubernetes.io/force-ssl-redirect: true
ingress.kubernetes.io/secure-backends: true
kubectl.kubernetes.io/last-applied-configuration: {"apiVersion":"extensions/v1beta1","kind":"Ingress","metadata":{"annotations":{"ingress.kubernetes.io/force-ssl-redirect":"true","ingress.kubernetes.io/proxy-body-size":"500m","ingress.kubernetes.io/secure-backends":"true","kubernetes.io/ingress.class":"nginx","kubernetes.io/tls-acme":"true","nginx.ingress.kubernetes.io/backend-protocol":"HTTPS","nginx.ingress.kubernetes.io/proxy-body-size":"500m","nginx.ingress.kubernetes.io/proxy-buffer-size":"16k","nginx.ingress.kubernetes.io/proxy-connect-timeout":"30","nginx.ingress.kubernetes.io/proxy-read-timeout":"1800","nginx.ingress.kubernetes.io/proxy-send-timeout":"1800","nginx.ingress.kubernetes.io/ssl-passthrough":"true"},"name":"vsystem","namespace":"datahub24"},"spec":{"rules":[{"host":"poc1.datahubpoc.com","http":{"paths":[{"backend":{"serviceName":"vsystem","servicePort":8797},"path":"/"}]}}],"tls":[{"hosts":["poc1.datahubpoc.com"],"secretName":"vsystem-tls-certs"}]}}
kubernetes.io/ingress.class: nginx
kubernetes.io/tls-acme: true
nginx.ingress.kubernetes.io/proxy-body-size: 500m
nginx.ingress.kubernetes.io/proxy-send-timeout: 1800
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal CREATE 26m nginx-ingress-controller Ingress datahub24/vsystem
Normal UPDATE 26m nginx-ingress-controller Ingress datahub24/vsystem
Note AWS has automatically assigned a classic load balance with A-Record : adb<......>.ap-northeast-2.elb.amazonaws.com
Check this at the AWS console under load balances:
Lets now create a CNAME in route 53 and assign poc1.<...>.com to the load balancer DNS name(A record) :
Wait for a few minutes for the DNS change to propagate or if you are too eager change TTL(seconds) to 60 in the record set. Access the application as : https://poc1.<...>.com 🙂
Note that the URL is already https secured as we had kube-lego in the steps above to call LetEncrypt to SSL secure the DNS_NAME defined in the Ingress YAML.
Including once more screenshot, as this is my favorite screen:
Overall its fairly easy to install SAP DATAHUB, the complexity often arises with the use of technologies such as docker , kubernetes but that's more of a learning curve.
- SAP Managed Tags:
6 Comments
