Automating regression tests with Docker

We recently started a project to create a new UI5 gui for one of our main products. We tried to reuse as much of the existing ABAP code and use a typescript application to make it easier to consume from the UI. As a bonus we’ll get a comprehensive rest API with an openAPI definition.

Continuous integration and delivery

All development is versioned in git with a nearly full CI/CD pipeline, and we have a pretty decent test coverage.

I say nearly full because ABAP changes are distributed to the relevant SAP instances with transports,  and this doesn’t trigger any pipeline.

We do merge them in the main repo with pull requests, like the rest of the code, and that does trigger a pipeline, but runs against the ABAP dev environment, not the main branch.

Also the CD part delivers to QA  and a demo environment, not production, so it’s not a real CD even if technically similar.

Regression tests

Until recently we had a big gap on integration tests. The application requires a relatively complex landscape, as one of its main jobs is to move transports:

Ideally we want to:

• run a regression test after every build
• on the latest version of the code
• everything else should be in the same exact status, including transports and log files

To achieve this, we created a docker container for each of the SAP instances and one for the node server, and an orchestrator service to easily backup/restore/replicate them.

Each test engineer has a playground which is a copy of the main landscape, where they do most of their work.

These can be recreated restoring a backup of the template with a simple command line utility, in about 15 minutes.

They also create test data in the template, and whenever they see fit they take a new snapshot with the same utility. This takes about 50 minutes, during which the template is not available.

Whenever some code is merged on the main branch, bitbucket will trigger a jenkins build which will:

• start the test runner instance
• refresh it from the latest backup of main
• install the latest version of the node/gui code
• run the regression tests in a headless browser
• shut down the test runner

A note about ABAP

For updating the ABAP code we still rely on transports, imported in the template with manual supervision. Would be possible to do it directly in the test runner with abapGit, but is tricky to handle the customizing part and a bit less reliable. Would also be possible to automate transport imports: we can cross reference transports with commits via bitbucket, jira and activecontrol.

But we decided it wasn’t worth the hassle as new ABAP is probably 3 to 5% of the development effort for this project.