Customized SAP Data Intelligence Deployment cycle using Github Action Workflows

Introduction:

This is going to be my first blog on SAP Data Intelligence and quite excited about sharing with all my SAP colleagues/members working on this tool.

This document overviews about deployment life cycle management for SAP Data Intelligence using CI/CD methodology. Since SAP DI doesn’t provide any proprietary tools for deployments as of the current DI version 2303.15.28.

Business Pain Points:

• There is no SAP proprietary solution for deployment of SAP DI pipelines and operators using CI/CD approach
• Exporting and Importing files into Production systems are restricted in some of the client deployments
• Manual method of deploying DI objects is susceptible to human mistakes

Business Value Proposition:

• Implementation of CI/CD throughout Development, testing and Production phases of SAP DI development.
• Automatic deployment of DI pipelines and operators without any manual intervention
• Review of the DI objects when pushed for deployment to check its completeness and inform developers/object owners about it.
• Authentication based approach where the DI objects are reviewed and authenticated at every level to discard any unintended changes to get moved between different systems.

Deployment Workflow Path:

The following picture shows the path how the objects are moved starting from development till getting it deployed into Production.

The following points to be noted here:

• Develop, Test and Master branches are protected so that no one can able to push to it directly.
• The changes can be moved from Feature branches to develop, test and Master separately which helps us independently deploy DI objects in case required.
• Each workflows are event based triggered like based on pull or push requests
• Each workflow can identify and deploy only new DI pipelines or changes pipeline rather than picking all from the repository. This is the strength of this pipeline which avoids re-deployment of unchanged DI objects into test and production systems.

Design of deployment workflow

The workflows are designed using Git Action commands. These action commands are branch specific and execution is based on event (Push or Pull requests). This workflow can identify the DI pipelines/operator files which are modified or newly created and trigger deployment only for those objects.

The sample code here can be referenced for designing your deployment workflows.

name: Deployment of DI objects to DEVELOPMENT tenant

on: 
  push:
    branches: [<Github development branch name>]

jobs:
  setup:
    runs-on: ubuntu-latest
    outputs:
      matrix: ${{steps.list_file.outputs.value}}
      
    steps:
      - name: check out repo
        uses: actions/checkout@v2

      - name: Get Diff Action
        uses: technote-space/get-diff-action@v6
        with:
          format: JSON

      - name: Filter files for graphs and operators only  
        run: |
          res=$(echo '${{ env.GIT_DIFF }}' | jq '.[] | select(. | contains("graph.json") or contains("operator.json"))' | jq -Rsc '. / "\n" - [""]')
          echo "filtered_op=$res" >> $GITHUB_ENV
        shell: bash

      - name: Passing the final output to build
        id: list_file
        run: |
          res=$(echo '${{ env.filtered_op}}' | jq '.[]')
          if [ -z $res ]; then
            echo '::set-output name=value::["NO DEPLOYMENT OBJECTS FOUND"]'
          else 
            echo '::set-output name=value::${{ env.filtered_op }}'
          fi        
        
  build:
    needs: [setup]
    runs-on: windows-latest
    strategy: 
      max-parallel: 1
      matrix: 
        value: ${{fromJson(needs.setup.outputs.matrix)}}
   
    steps:
      - name: Checking out your current git repository
        if: ${{ matrix.value != 'NO DEPLOYMENT OBJECTS FOUND' }}
        uses: actions/checkout@v2

      - name: Capturing the solution name to bundle
        if: ${{ matrix.value != 'NO DEPLOYMENT OBJECTS FOUND' }}
        run: echo "obj_name=$(echo ${{ matrix.value }} | awk 'BEGIN{FS=OFS="/"} {print $--NF}')" >> $GITHUB_ENV
        shell: bash

      - name: Finding the Source directory
        if: ${{ matrix.value != 'NO DEPLOYMENT OBJECTS FOUND' }}
        run: |
          echo "source_dir=$(echo ${{matrix.value}} | awk 'BEGIN{FS=OFS="/"}NF--')" >> $GITHUB_ENV
        shell: bash
                
      - name: Defining the target directory for solution
        if: ${{ matrix.value != 'NO DEPLOYMENT OBJECTS FOUND' }}
        run: |
          echo "target_dir=$(echo ${{ env.obj_name }}/content/files/vflow/${{env.source_dir}})" >> $GITHUB_ENV
        shell: bash

      - name: Building the solution directory structure 
        if: ${{ matrix.value != 'NO DEPLOYMENT OBJECTS FOUND' }}
        run: | 
          mkdir -p ${{env.target_dir}}
          cp -R ${{env.source_dir}}/* ${{env.target_dir}}/
          cp -R ${{env.source_dir}}/manifest.json ${{ env.obj_name }}/
        shell: bash

      - name: Log-in to DI through vctl
        if: ${{ matrix.value != 'NO DEPLOYMENT OBJECTS FOUND' }}
        run: vctl\vctl.exe login ${{secrets.<DEV INSTANCE URL GIT SECRET>}} --user-cert ${{secrets.<bundle.pem GIT SECRET>}} ${{secrets.<key.pem GIT SECRET>}}

      - name: Bundle the solution into zip file in current working directory
        if: ${{ matrix.value != 'NO DEPLOYMENT OBJECTS FOUND' }}
        run: vctl\vctl.exe solution bundle ${{env.obj_name}}

      - name: Capture the solution version post bundle
        if: ${{ matrix.value != 'NO DEPLOYMENT OBJECTS FOUND' }}
        run: |
          echo "zip_name=$(ls | sed -n 's/\.zip$//p')" >> $GITHUB_ENV
        shell: bash

      - name: Get list of solutions from the strategy 
        if: ${{ matrix.value != 'NO DEPLOYMENT OBJECTS FOUND' }}
        run: |
          vctl\vctl.exe strategy get ${{secrets.<default DI strategy name in GIT SECRET>} -o "json" | Out-File ./input.json
      
      - name: Look up for this solution version if available in the strategy
        if: ${{ matrix.value != 'NO DEPLOYMENT OBJECTS FOUND' }}
        run: |
          jarray=$(cat ./input.json | jq '.layers')
          obj_arr=$(echo $jarray | jq -c -r '.[]')
          echo "str_exist_flag=$(echo "NO")" >> $GITHUB_ENV
          flag=0
          for obj in ${obj_arr[*]}
          do
            val=$(echo $obj)
            if [ "$val" = "${{env.zip_name}}" ]; then
              echo "str_exist_flag=$(echo "YES")" >> $GITHUB_ENV
              flag=1
            fi
          done
          if [ $flag -eq 1 ]; then
            echo "This solution ${{env.zip_name}} version already exists in strategy."
          else 
            echo "This solution ${{env.zip_name}} version doesn't exists in strategy."
          fi
        shell: bash

      - name: Remove the solution version from strategy if already exists
        if: ${{ env.str_exist_flag == 'YES' }}
        run: vctl\vctl.exe strategy remove ${{secrets.<default DI strategy name in GIT SECRET>}} ${{env.zip_name}}

      - name: Get list of all solutions from the tenant 
        if: ${{ matrix.value != 'NO DEPLOYMENT OBJECTS FOUND' }}
        run: |
          vctl\vctl.exe solution list -o "json" | Out-File ./solution.json
      
      - name: Look up for this solution version if available in the tenant
        if: ${{ matrix.value != 'NO DEPLOYMENT OBJECTS FOUND' }}
        run: |
          obj_arr=( $(jq -c -r '.[] | (.Manifest.name) + "-" + (.Manifest.version)' solution.json) )
          echo "sol_exist_flag=$(echo "NO")" >> $GITHUB_ENV
          flag=0
          for obj in ${obj_arr[*]}
          do
            val=$(echo $obj)
            if [ "$val" = "${{env.zip_name}}" ]; then
              echo "sol_exist_flag=$(echo "YES")" >> $GITHUB_ENV
              flag=1
            fi
          done
          if [ $flag -eq 1 ]; then
            echo "This solution ${{env.zip_name}} version already exists in solution list."
          else 
            echo "This solution ${{env.zip_name}} version doesn't exists in solutions."
          fi
        shell: bash

      - name: Delete the existing solution version from tenant if already exists
        if: ${{ env.sol_exist_flag == 'YES' }}
        run: vctl\vctl.exe solution delete $(echo ${{env.zip_name}} | cut -d "-" -f 1) $(echo ${{env.zip_name}} | cut -d "-" -f 2)

      - name: Uploading updated solution version to the tenant
        if: ${{ matrix.value != 'NO DEPLOYMENT OBJECTS FOUND' }}
        run: vctl\vctl.exe solution upload ${{env.zip_name}}.zip

      - name: Adding updated solution version to the strategy
        if: ${{ matrix.value != 'NO DEPLOYMENT OBJECTS FOUND' }}
        run: vctl\vctl.exe strategy add ${{secrets.<default DI strategy name in GIT SECRET>}} ${{env.zip_name}}

Expected learnings from this blog:

• This is a ready to use template which can be reused in DI project deployments with CI/CD approach.
• With the use of this approach, manual handling of deployments like import/Export can be avoided.
• We can track the history of changes for any object using git commit history.

I encourage all readers to provide your comment and happy to hear back any suggestions/ improvements in this approach. This will help me to improve this deployment workflow.