Continuous Integration in ABAP using Jenkins
In my recent project, we are building Java miniservices that connect to SAP backend via RESTful APIs. The team has setup a CI/CD pipeline in Jenkins to automate build, test, and deploy of those Java services. At the time we had the pipeline, I have realized the benefits for having such pipeline and that became my motivation to build one for ABAP.
The most important benefit in my view is the visibility over your software’s health. If someone asks whether the Java piece is working fine at the moment, we can have a glance at Jenkins monitor and be able to answer quite confidently. In ABAP, we are just blind. I even sometime break my software and found them out later.
I think it’d be good if I have one for ABAP so at least when somethings goes wrong, I, as ABAP developer, should know it as soon as possible before someone else does. I didn’t expect much at the beginning but just to execute some tests every time we make changes in ABAP i.e. continuous integration.
Who know, I eventually got both CI and CD in the pipeline for ABAP running. Not perfect, but good enough. Here is the pipeline in Jenkins for ABAP:
Comparing to our Java pipeline:
In this blog, I will guide how to setup the same CI pipeline for your ABAP backend.
Prerequisites
- An ABAP system with ADT (ABAP in Eclipse) enabled and abapGit installed
- A Jenkins server with Node.js and newman installed
My recommendation is to build a SAP NetWeaver AS ABAP and SAP BW 7.50 SP02 on ASE [Developer Edition] on the cloud either Azure or AWS via SAP CAL. If you have no idea how to do this then you may check out this blog for instructions. It will be a bit different as some steps are getting easier now.
For Jenkins, once you get the Developer Edition up and running, you also get a Windows 2012 server frontend instance as well. Download and install Jenkins on the server. Install all recommended plugins and you should be ready to roll.
For Node.js, you can use Chocolatey to ease the installation. For newman, you can install via npm (comes with Node.js).
Using Git Repository for ABAP
Why do we use Git for ABAP? This is the question I usually get. Well, I didn’t want to use at the beginning but I want to deploy a mechanism where the Jenkins is triggered every time when ABAP code is changed.
I may insert a custom logic somewhere in ABAP to trigger Git when developer release their task which similar to when Java developer commit and push their code. But this is too complicated. I don’t want to write ABAP codes to automate ABAP codes.
Since there’s great tool like abapGit and I know Git can trigger Jenkins then why don’t we do them to ease our works.
Later, I found more benefits using Git for ABAP. One day, I was asked what changes over the past 2 weeks we are moving to production. I made lots of changes during those 2 weeks so I can’t remember all of them. I just go to Git and see all the logs of changes I made. I can even show exactly what changes I had made line by line.
Apart from change log, another benefit for developer is code backup. I sometime need to restore to the code I wrote and already deleted yesterday or last week. It usually a problem for me when I haven’t started using Git because, by nature, ABAP developers are not create new version that often. Using Git and apply practice ‘commit every day’ helps a lot.
Git also helps enforcing developer to modularize their objects in separated package. I used to see more than 3,000 objects in the same package then package is not used by its purpose at all.
Pushing Codes to Git
If you haven’t done so, create a Git repository and push your ABAP codes into it. In this tutorial, I’ll use my simple REST API repository as an example. Feel free to clone it to package the $REST_SIMPLE.
Adding Jenkinsfile
Go to your Git repository and add Jenkinsfile (You don’t have to do this if you use my repository as I already did). Paste the following content and commit.
You should see Jenkinsfile in your repository.
Building Jenkins Pipeline
You can create job in Jenkins one-by-one to automate each step but in this tutorial I will use Jenkins pipeline script in Apache Groovy format which is easier to reuse and maintain.
The good thing about PaaC (Pipeline as a Code) like groovy is you can maintain the whole pipeline in the code format that mean you can version-control it on Git as well. Coding means less configuration required and minimize clicking, typing, and waiting through the UI.
Creating a New Pipeline Job
In Jenkins, create a new Pipeline job.
Set up Build Triggers to pull from Git every minute. Or you may setup web hook.
Configure Pipeline by reading Jenkinsfile from your Git repository.
Create a Credential
Go to Credentials and add a new one like below. This is the username and password you will let Jenkins to log on to ABAP system.
Run
Go back to your job and click Build Now to start building. If everything is setup correctly, you should see the pipeline completed successfully like below.
Continuous Delivery
Continuous Delivery is about keeping delivering value to customer. Features or changes that are done should be moved to QA/Staging as soon as possible so user can test and provide feedback.
You can append Jenkinsfile to perform additional stages to get your changes into QA environment. This will vary depending on process and system landscape in your company. How to write Jenkinsfile is another long blog so I won’t explain it here.
In my case, I’ve decided to build the tools to automate our current process. We are using SolMan to manage transport. Normally, when we want to move change to QA, we log on ChaRM BSP webpage and release the document from there. Our import job runs regularly to import changes into QA.
So I built an UI automation to automate this manual process. Wait for transport import. Then run another set of tests on QA. The tool I used is Robot Framework and Selenium. Of course, that can be another long blog.
However, in my case, there’s still a gap in this pipeline. Because ABAP developer can skip this quality pipeline and go back to their traditional steps. So I view this pipeline as an option to help developers and the team delivery faster with more quality. If we need to enforce the pipeline, then that would require a big change in the process and the way of work for many teams.
I hope this will benefit or at least motivate you to improve the way we work. Please share your comment about this. Let’s modernize ABAP!
Continue reading my next blog: Application Telemetry with Prometheus – learn how to monitor your SAP API with Prometheus and Grafana!
Thanks for sharing!
One question: For me it's not quite clear how you achieved that the Jenkins CI server executes the ATC checks and unit tests on the ABAP system. I saw the jenkins file in the repo and the JSON files that newman seems to need, is that all? I assume there must be a HTTP path and a SICF node to remotely execute ATC things over HTTP, right? And how is the result retrieved and converted to a format that Jenkins can display? Is that programmed by you on the ABAP side of things or is there an official API available for that?
Hi Fabien,
Yes, there are APIs available to do that. The same APIs that used by ADT (ABAP in Eclipse). I use Newman for calling those APIs and check the results (by Javascript). Please check out this blog for more detail.
Great, thanks. Exactly what I was looking for.
I got another question: Am I correct in the assumption that not the committed code is tested using ATC but rather the active one in SAP at the time of build pipeline execution?
I. e. you changed 3 source files and activated them but only committed one of them to the repository while the other 2 are still staged but not committed. The test coverage result for example will show the result of the 3 changed source files and not of the current state in the repository.
Or you just committed your files and changed the branch in abapGit to work on a different problem. The CI checks will then use the source code of the "new" branch which has nothing to do with the state of the repository that was committed previously.
Or is there something I am missing?
Hi Fabien,
Yes, your assumption is right. ABAP Unit and ATC are run against the active version in ABAP system, not the ones you commit to Git. As Git is not the main SCM/SVM for ABAP so I don't want to screw up the way people work now in term of change and transport management.
I better wait to see how Git will work for ABAP in the cloud before making further experiment.
Maybe one could set up a centralized ATC system that git pulls the repository state under test automatically, runs the checks, reports back to jenkins and removes the repository again afterwards?
But I agree, waiting on some details on the SAP git integration is a good idea.
I also think about the ability to spin up an ABAP instance on-demand to run tests like the we do with container. But that seems to be hard as ABAP platform is so big and we may need Production data as well to run API+UI tests as well.
Like you mentioned, centralized system seems to be more practical but I would need to keep data refreshed from Production regularly anyway.
Another (crazy?) idea is to use the ADT api to create a temporary package with the whole code from the commit and test it there.
Chairat Onyaem Great Posts btw :D! I'm definitely gonna play around with it.
Another idea I heard is to clone the production system into ‘pre-production’ system, transport and test your codes there. I think about create the whole system on-demand like we do with containers. But that seem to be impractical for the big system like SAP. So having one extra environment and keep it in sync with production may be more practical.
Well done Chairat!
Thank you.
Great blog, thank you
I'am very intrested in the combination charm and the ci pipline you describe here.
We are in the progress of implementing charm and i wonder if we need that if we using a full ci/cd process. Maybe sap is not completly ready for this but now having a kind of toll-gate system in charm seems to be contradicting what you would like in ci/cd
I would really like you inside on this.
I don’t think this can replace charm as charm is standard change control process from sap and sap system is so unique and is not designed to be continuous delivery. I see this can be a special process that can help speed up charm for specific module/project where the app will be designed to be more continuous delivery.
I wish charm has APIs so we can easily integrested with any ci/cd tool of choice.
With Solution Manager 7.2 SP6 there are some REST APIs comming, which you could use to handle change management actions, e.g. create transports and get a lot of information about transports.
This github project provides a command line client for this interface. https://github.com/SAP/devops-cm-client
check it out!
Hi Chairat,
Is any customer using this process in their production environment?
Hi, I don't aware of anyone who actively using this as the main process and I don't recommend either.
Hi Chairat,
Thank you for a very informative blog. We've setup runnint abap unit tests from Azure Devops via your Newman script, and it's working fine.
However we're "missing" lots of information that would be good to have, like a ADT-link to the object and so on, do you know if it's possible to add more to the junit report?
I assume it's a limitation in newmans test-framework, that it cannot produce a full junit report. A shortcut I guess would be to concatinate information into the test-array
My long-term approach is to develop a new/small application (probably with Golang) to execute unit tests and generate proper JUnit report. Using newman is still too complicated to run in a pipeline (i.e. needs Node.js Jenkins slave instance). I can't promise when I gonna complete. Still too busy these days. 🙂
For short-term, yes, you can put that in the newman test name.
In the end I skipped Newman and took the XSLT route. I've created a NodeJS script that triggers Abap Unit tests (via the ADT rest service) and then transforms them via XSLT into JUnit format. We're running it on Azure Devops with no issues.
I'll check if I'm allowed to open source it and share.
I did publish a blog describing our approach. Still needs Node, but that's what I'm mostly comfortable with.
https://blogs.sap.com/2018/12/17/abap-continuous-integration-with-azure-devops-abap-unit-in-the-cloud/#
Hello Chairat,
Thank you for your insightful blog. Your blog has helped us a lot, for finding our first phase of working with CI/CD with abap on jenkins. But we are so much curious to know more that how it will going to scale much. How we can add more functionality or how it can help us to work with,
Multiple TR case, Multiple ABAP object case and Multiple Pipeline case.
We are also looking forward for the blog post by Mattias Johansson , let's see where it will go.
Waiting for more informative blogs and results.
Thanks