Skip to Content
Technical Articles

CI/CD Tools for SAP Cloud Platform ABAP Environment

In the on-premise ABAP world, lifecycle and quality processes can be controlled by the ABAP systems themselves. Think of transport routes and jobs to execute different checks. In the SAP Cloud Platform ABAP Environment aka Steampunk, this is no longer possible. Steampunk systems are independent of each other, and source code is “transported” with git-based solutions: gCTS and abapGit. While different systems in one global account are connected to the same repositories (read: “software components”), they do not know of each other. Due to this, the driver of lifecycle and quality processes needs to shift from the ABAP system itself to an external driver – for example: a classic Continuous Integration / Continuous Delivery (CI/CD) server like Jenkins. Of course, the Steampunk systems need to expose the functionality via APIs so that a CI/CD server can control those processes.

Components%20of%20Jenkins%20driven%20Steampunk%20CI/CD%20processes

Components of Jenkins driven Steampunk CI/CD processes

In the next sections, I want to introduce the released APIs regarding gCTS and ABAP Test Cockpit (ATC), as well as all the functionality we built on top of these APIs to make your life easier: reusable functions using these endpoints and a predefined pipeline which lets you easily create CI/CD processes for Steampunk.

To make the consumption of our pipeline and functions (or “steps” in the context of a pipeline) as convenient as possible, we decided to contribute them to the Project Piper. This open-source project is the home of different steps and pipelines related to the SAP Ecosystem.

Let’s start with the easiest way to create a pipeline and drill our way down to the APIs.

 

ABAP Environment Pipeline

The “ABAP Environment Pipeline” is a predefined pipeline usable on a Jenkins server. It covers most use cases and is ready to use: no programming is required – only the configuration of the pipeline is necessary. The pipeline consists of several stages. A stage groups a set of steps together, which are related in content. One stage, for example, is called “Prepare System”. As the name indicates, this stage takes care of preparing a system that is usable within the pipeline. It includes creating a Steampunk system and configuring the communication.

Layers%20of%20a%20predefined%20Jenkins%20Pipeline

Layers of a predefined Jenkins Pipeline

The ABAP Environment Pipeline supports various use cases. A simple scenario that can be realized with this predefined pipeline is running nightly ATC checks on the released source code. (Of course, you will benefit most from such a pipeline if you release your Transport Requests early and often.) This pipeline would consist of the following stages:

  • Prepare System: This stage creates the Steampunk system and configures the communication to the system
  • Clone Repositories: This stage clones your repositories (software components) to the system
  • ATC: This stage executes ATC checks on the configured repositories
  • Post: This stage is automatically executed in the end and will delete the Steampunk system

 

Diagram%20of%20the%20ABAP%20Environment%20Pipeline.%20The%20Post%20stage%20is%20not%20displayed.%20Stages%20that%20are%20not%20configured%20are%20skipped.

Diagram of the ABAP Environment Pipeline. The Post stage is not displayed. Stages that are not configured are skipped.

 

While the “ABAP Environment Pipeline” itself is predefined, you still have the option to customize it via configuration and extension. In principle, a stage of this pipeline will only be executed if it is configured. This means that, if I do not want to create a new Steampunk instance for every pipeline execution, I just do not configure the “Prepare System” stage. In case a specific stage does not quite fit my needs, I can create an extension file for this stage and add missing functionality (or even rewrite the whole stage). A popular example is to extend the ATC stage with the “CheckStyle” Plugin, which creates helpful diagrams out of the XML output of the ATC API.

Example%20set%20of%20CheckStyle%20diagrams

Example set of CheckStyle diagrams

As you can imagine, there is a lot of room for creativity. You can head to the documentation to learn more about the pipeline or check out some examples in this repository.

If this pipeline does not quite fit your needs, you still have the option to reuse our steps to create your own pipeline. Let’s have a look in the next section.

 

Pipeline Steps

I want to start this section by highlighting the pipeline steps’ design pattern in Project Piper. Some years ago, Piper’s focus was on the CI server Jenkins: The pipeline steps were written in the language “Groovy” and could only run on this technology. Nowadays, while Jenkins is still widespread, other technologies are gaining popularity. Therefore, the decision was made to switch the implementation of pipeline steps over to “Go” (or “Golang”). This language compiles the source code to a binary, which can be executed on different CI servers. Here is an example of how the “clone” step can be called via the piper binary:

abapEnvironmentCloneGitRepo --host host.com --username myUser --password myPassword --repositoryName  “/DMO/GIT_REPOSITORY” --branchName master

Of course, for users of a Jenkins CI server, a Jenkins step implementation is built on top of the Golang implementation. So, the full picture is as follows: several WebAPI endpoints of Steampunk are called by the Golang implementation and the resulting binary is then used by the Jenkins step.

Layers of a Piper step

 

Before looking at the WebAPI, let me show you an example of a Jenkinsfile defining a custom pipeline and reusing the Jenkins step “abapEnvironmentCloneGitRepo”.

@Library(['piper- lib-os']) _
	
pipeline {
  agent any
  stages{
    stage("Pull Git Repository"){
      steps {
        abapEnvironmentCloneGitRepo(script: this) 
      }
    }
  }
}

The annotation “@Library([‘piper-lib-os’]) _” is a reference to the Project Piper jenkins-library (https://github.com/SAP/jenkins-library/), which would have been configured on the Jenkins server with the ID ‘piper-lib-os’. The reused step is part of a pipeline stage and simply called with the statement “abapEnvironmentCloneGitRepo(script: this)”. The parameters for this step (e.g. endpoints, credential IDs etc.) would be provided in a separate configuration file.

The documentation of our example step for the action “Clone” can be found here. Other available steps can then be found in the documentation menu under “Library steps”.

Now that you know how to use the steps in Jenkins or via the Golang binary, what are they actually doing? Let’s have a look at our example step: the action of cloning a repository. At first, the step authenticates on the system (and fetches the X-Csrf-Token, which is required for a POST request). After triggering the “clone” action, the step polls the status of the action to find out, when the repository is finally cloned successfully. As you can see, the steps implement a simple series of API requests to achieve a single task (“clone a git repository”).

Of course, if you really want to control every detail of your pipeline (or want to run it on a CI server that is not able to support the Golang binary), you can use the released APIs for gCTS and ABAP Test Cockpit directly. Let’s have a look at them in the next section.

APIs

The goal of the gCTS API is to enable you to automate the most crucial functions that you may know from the “Manage Software Components” app: cloning a repository, pulling a repository, and checking out a branch. The service, including those endpoints, is documented here.

For example, the following request would clone the software component “/DMO/GIT_REPOSITORY” to the system, with the branch “master” being checked out.

 

POST/sap/opu/odata/sap/MANAGE_GIT_REPOSITORY/Clones HTTP/1.1
Content-Type: application/json
Accept: application/json

{
        "sc_name": "/DMO/GIT_REPOSITORY"
        "branch_name": "master"
}

 

A second service would then allow you to execute a check run with the ABAP Test Cockpit (ATC). You can specify the check variant and the software component or package to be checked in the request body. Here is an example:

 

POST https://<host.com>:<port>/sap/bc/adt/api/atc/runs?clientWait=false
Content-Type: application/vnd.sap.atc.run.parameters.v1+xml

<?xml version="1.0" encoding="UTF-8"?>
<atc:runparameters xmlns:atc="http://www.sap.com/adt/atc" xmlns:obj="http://www.sap.com/adt/objectset">
  <obj:objectSet>
      <obj:softwarecomponents>      
        <obj:softwarecomponent value="/DMO/GIT_REPOSITORY"/>
      </obj:softwarecomponents>
   </obj:objectSet>
</atc:runparameters>

 

The full documentation can be found here.

 

What’s next?

In the future, the ABAP Environment Pipeline will support the build process of ABAP Add-ons for Steampunk. This scenario is intended for SAP partners who want to build Software as a Service (SaaS) solutions for the SAP Cloud Platform. Some stages and steps are already contributed and part of the ABAP Environment Pipeline. While this is currently our focus, we are still continuously improving the existing pipeline and steps, and we are hoping to support more tools in the future.

Please let me know if you have feedback or questions and feel free to post in the comment section below (or head to the Answer Hub for general questions).

 

Notice: images are my own.

8 Comments
You must be Logged on to comment or reply to a post.
  • Thanks, few questions/comments,

     

    A: Can this be tested in the ABAP Steampunk trial systems?

    B: According to https://sap.github.io/jenkins-library/pipelines/abapEnvironment/introduction/#prepare-system, the system is not created automatically ? Following [1] the automation part is important for CI/CD, I guess this will be fixed in a upcoming release?

    D: What would the typical flow/trigger/lifecycle be of this pipeline? With one development system I guess there will not be feature branches? So the integration will not be for “each person”/”each integration” [1], but a set of integrations by a set of developers?

    C: for “Executing ABAP Test Cockpit (ATC) Check Runs”,  https://help.sap.com/viewer/65de2977205c403bbc107264b8eccf4b/Cloud/en-US/d8cec788fc104ff9ad9c3757b4dd13d4.html , it looks like it is not possible to configure the variant? Or am I missing something?

    E: “Keep the build fast” in [1], any experiences on the time taken to execute the pipeline?

     

    [1] https://martinfowler.com/articles/continuousIntegration.html

    • Thanks for your comment!

      Regarding your questions:

      A: I don’t think this is possible as both the software component lifecycle and the communication is restricted (see this blog post). 

      B: The system itself is created automatically, but we have currently no easily accessible API to check when the system is ready. But, hopefully this will be solved soon.

      D: Branches are generally supported. Currently, they are intended to be used as “release branches”, e.g. to split up the development into a “development” and “maintenance” code line. Personally, I’m a big fan of “feature branch” development (or via forks), but I think there are a few missing pieces to do this with Steampunk systems. For now, it is also only possible to have a scheduled pipeline.

      So, to answer your question: the typical case would be a single development system and a scheduled pipeline supplying different systems with the released development and/or running checks.

      C: As far as I know, this should be possible (see last example in this documentation).

      E: The most time consuming part is creating a Steampunk system (see also point B). When this can be done fully automated – hopefully soon – I would estimate that the pipeline takes about one hour to run (probably a bit faster). It is, of course, also possible to use a static Steampunk system. Then, I assume, the deciding factor would be the size of the software component. Our tests (with small software components) took around five minutes.

      • Thanks, sorry about the letter sequence(ABDCE), but guess its easier to keep it that way 🙂

        D: Does this mean Steampunk will, in the future, support having one development system where each developer can work on their own branch within the same software component at the same time?

        I think there is a large difference between having a scheduled job running and Continuous Integration, with a scheduled job it is difficult to figure out where some changes broke, assuming multiple developers have made changes since last scheduled job.

         

        E: Its difficult, a static system could potentially produce incorrect results, eg. with unit tests modifying database contents or calling XCO framework. Its not a correct unit test if it does this, but in a generic setup it might be worth making some compromise depending on the nature of the development.

        from [1],

        Most of my colleagues consider a build that takes an hour to be totally unreasonable

        Another compromise, not sure if its possible(?), would be to have systems on stand-by, ready to accept pipeline jobs.

        And also front load as much as possible to run before it reaches an actual ABAP system, see “Scenario 9” in [2]

         

        [1] https://martinfowler.com/articles/continuousIntegration.html

        [2] https://blogs.sap.com/2020/02/20/fully-supporting-ci-in-abap-as/

         

        • D: Does this mean Steampunk will, in the future, support having one development system where each developer can work on their own branch within the same software component at the same time?

          No, I did not want to imply that.

          I think there is a large difference between having a scheduled job running and Continuous Integration, with a scheduled job it is difficult to figure out where some changes broke, assuming multiple developers have made changes since last scheduled job.

          I agree.

          Another compromise, not sure if its possible(?), would be to have systems on stand-by, ready to accept pipeline jobs.

          I think this could be possible. But this is not planned on our side and has to be done by the consumer. This would be a reason to choose a custom made pipeline (and only reusing the steps) over the predefine ABAP Environment Pipeline.

  • Great to see things moving forward towards a CI/CD approach! For sure there will be several technical challenges to fully get there. Keep it up!

    Not to mention all other barriers that typically relegates technical excellence to a lower role. Or the huge skills set (and mindset) gap that makes it possible and relevant in a customer code base.

    Anyway! I had a question: what is the ideia for running AUNIT tests in the pipeline, through some ATC check? Sorry if it is a dumb question, I still didn’t.go through all the documentation.

    Regards,

    Felipe S.