Automated test execution is crucial in any DevOps process. Therefore, Azure Pipelines offers an integrated functionality to execute tests and provide a view to analyze the results as it can be seen in the screenshot below. To make it work with an UI5 application, you need to setup a testrunner that executes implemented integration and unit tests. In this blog post, we use Karma testrunner to execute pre-defined tests and publish the results in Azure Pipelines.
The blog post is divided into three parts. In the first part, the setup of the Karma testrunner is described. The second part explains the configuration of Azure-Pipelines to view test results. In the last part, the results are shown. The used source code in this blog post can be found on GitHub ( QP_UI5_Karma_Testrunner).
At first, we need a UI5 application with few tests defined. As we use a simple UI5 application with no semantic functionality, we also use trivial tests. As it can be seen in the subsequent screenshot, we have defined two QUnit tests. Both tests are placed within the test folder of the UI5 application.
Next, we configure Karma to execute the defined tests. Note, Karma is only a testrunner tool and can not be seen as a testing framework. At first, we need to install Karma and required modules via NPM. A list of required npm modules is defined in the package.json and can be seen below. If you use the repository linked to this blog post, you just need to execute ‘npm install’.
Next, to start the tests with “npm test”, you need to add the script “test” and map it to “karma start”. With this, every time you perform “npm test” from the command line, the karma runner starts and executes all tests defined within the UI5 application.
In the last step you need create a karma configuration. In order to do so, create a file named “karma.conf.js”. This is the default name for the karma configuration. If you use a name other than the default name, you need to provide the name in the “karma start” configuration.
The configuration is quite small but there are some pitfalls. Since it is an UI5 application, we use the “UI5” framework of Karma to execute tests. In the UI5 section, we give the location of the UI5 libraries. From the Karma documentation (Karma UI5 documentation) this configuration is not needed if a ui5.yaml file is present but this never worked for us. In the next section, three reporters are defined. The first ‘progress’ is just for convenience and prints the test results to the command line. The second ‘junit’ executes the defined unit tests and the last ‘coverage’ measures the code coverage.
In the next two sections ‘preprocessors’ and ‘coverageReporter’ the coverage reporter is configured. We use cobertura since this output format can be parsed and visualized in Azure-Pipelines. Finally, we configure Karma to use a headless Chrome browser and to only execute tests once. A headless browser is required so Karma does not open a real browser and to execute the tests in the background.
To test Karma, we need to execute ‘npm test’ or ‘karma start’ from the command line. If all works correct two files, TESTS-*.xml and cobertura.xml, should be created:
The first file contains the results of the test execution, the latter contains information about the achieved code coverage. Next, we parse this output in Azure-Pipelines and visualize it.
As well as the Karma config, the Azure-Pipelines config is quite small and can be seen below. At first, a trigger is defined. This is the branch name of the connected GitHub repository. After the trigger, the pipeline runs four stages sequentially.
To test the setup, trigger the pipeline from Azure DevOps. After the pipeline has completed, you will see two additional tabs ‘Tests’ and ‘Code Coverage’. Within ‘Tests’ you see an overview and the ratio between successful and failing tests. If there are failing tests, you can drill down into more details and get the info which tests have failed.
Within ‘Code Coverage’ you get information about the code coverage which are executed during test. Further, you get a summary of the overall code coverage as well as the code coverage of each file.
This blog post explained how you can integrate test reporting to Azure DevOps with two small configurations. At first, we configured Karma to run tests of an UI5 application. At second, we prepared Azure-Pipelines to visualize the test results. The result showed a test summary in the Azure-Pipeline execution. This functionality helps us to continuously monitor automated test execution and is therefore key of an essential part of a DevOps process.
Michael Christa is SAP Consultant at Q_PERIOR focusing on technology and innovation. His professional interests are developing end-to-end SAPUI5 applications as well as working with the SAP Business Technology Platform.
Azure-Pipelines shows test results generated by Karma
The blog post is divided into three parts. In the first part, the setup of the Karma testrunner is described. The second part explains the configuration of Azure-Pipelines to view test results. In the last part, the results are shown. The used source code in this blog post can be found on GitHub ( QP_UI5_Karma_Testrunner).
UI5 application including Karma testrunner
At first, we need a UI5 application with few tests defined. As we use a simple UI5 application with no semantic functionality, we also use trivial tests. As it can be seen in the subsequent screenshot, we have defined two QUnit tests. Both tests are placed within the test folder of the UI5 application.
QUnit tests defined within the UI5 application
Next, we configure Karma to execute the defined tests. Note, Karma is only a testrunner tool and can not be seen as a testing framework. At first, we need to install Karma and required modules via NPM. A list of required npm modules is defined in the package.json and can be seen below. If you use the repository linked to this blog post, you just need to execute ‘npm install’.
"devDependencies": {
"@sap/ux-ui5-tooling": "1",
"@ui5/cli": "^2.11.1",
"@ui5/fs": "^2.0.6",
"@ui5/logger": "^2.0.1",
"karma": "^6.3.4",
"karma-chrome-launcher": "^3.1.0",
"karma-coverage": "^2.0.3",
"karma-junit-reporter": "^2.0.1",
"karma-ui5": "^2.3.4",
"puppeteer": "^10.2.0",
"rimraf": "3.0.2"
}Next, to start the tests with “npm test”, you need to add the script “test” and map it to “karma start”. With this, every time you perform “npm test” from the command line, the karma runner starts and executes all tests defined within the UI5 application.
"scripts": {
"start": "fiori run --open 'index.html'",
"test": "karma start",
"unit-tests": "fiori run --open test/unit/unitTests.qunit.html",
"int-tests": "fiori run --open test/integration/opaTests.qunit.html"
},In the last step you need create a karma configuration. In order to do so, create a file named “karma.conf.js”. This is the default name for the karma configuration. If you use a name other than the default name, you need to provide the name in the “karma start” configuration.
The configuration is quite small but there are some pitfalls. Since it is an UI5 application, we use the “UI5” framework of Karma to execute tests. In the UI5 section, we give the location of the UI5 libraries. From the Karma documentation (Karma UI5 documentation) this configuration is not needed if a ui5.yaml file is present but this never worked for us. In the next section, three reporters are defined. The first ‘progress’ is just for convenience and prints the test results to the command line. The second ‘junit’ executes the defined unit tests and the last ‘coverage’ measures the code coverage.
module.exports = function(config) {
config.set({
frameworks: ["ui5"],
ui5: {
url: "https://sapui5.hana.ondemand.com"
},
reporters: ['progress', 'junit', 'coverage'],
preprocessors: {
// Files that should be included or excluded from coverage analysis
'webapp/*.js': ['coverage'],
'webapp/!(test|localService)/**/*.js': ['coverage']
},
coverageReporter: {
type : 'cobertura',
dir : 'coverage/',
file: 'cobertura.xml'
},
browsers: ["ChromeHeadless"],
singleRun: true,
failOnEmptyTestSuite: false
});
};In the next two sections ‘preprocessors’ and ‘coverageReporter’ the coverage reporter is configured. We use cobertura since this output format can be parsed and visualized in Azure-Pipelines. Finally, we configure Karma to use a headless Chrome browser and to only execute tests once. A headless browser is required so Karma does not open a real browser and to execute the tests in the background.
To test Karma, we need to execute ‘npm test’ or ‘karma start’ from the command line. If all works correct two files, TESTS-*.xml and cobertura.xml, should be created:
Files generated by Karma testrunner
The first file contains the results of the test execution, the latter contains information about the achieved code coverage. Next, we parse this output in Azure-Pipelines and visualize it.
Azure-Pipeline Config
As well as the Karma config, the Azure-Pipelines config is quite small and can be seen below. At first, a trigger is defined. This is the branch name of the connected GitHub repository. After the trigger, the pipeline runs four stages sequentially.
- Install all required npm modules from the package.json
- Start the Karma testrunner in order to get test and coverage result files(TESTS-*.xml and cobertura.xml)
- Parse the unit test result (TESTS-*.xml) and visualize it in the pipeline tool. Note that you need to set the testResultsFormat to ‘JUnit’.
- Parse the cobertura test coverage (cobertura.xml) and publish the results as well.
trigger:
- main
pool:
vmImage: ubuntu-latest
steps:
- task: NodeTool@0
inputs:
versionSpec: '10.x'
displayName: 'Install Node.js'
- script: |
npm install
displayName: 'npm install and build'
- script: |
npm test
continueOnError: true
displayName: 'Run karma test'
- task: PublishTestResults@2
inputs:
testResultsFormat: 'JUnit'
testResultsFiles: '**/TESTS-*.xml'
displayName: 'Display Unit Test result'
- task: PublishCodeCoverageResults@1
inputs:
codeCoverageTool: 'Cobertura'
summaryFileLocation: '$(Build.SourcesDirectory)/coverage/**/cobertura.xml'Run the setup
To test the setup, trigger the pipeline from Azure DevOps. After the pipeline has completed, you will see two additional tabs ‘Tests’ and ‘Code Coverage’. Within ‘Tests’ you see an overview and the ratio between successful and failing tests. If there are failing tests, you can drill down into more details and get the info which tests have failed.
Azure-Pipelines shows test results generated by Karma
Within ‘Code Coverage’ you get information about the code coverage which are executed during test. Further, you get a summary of the overall code coverage as well as the code coverage of each file.
Azure-Pipelines shows coverage generated by Karma coverage reporter
Summary
This blog post explained how you can integrate test reporting to Azure DevOps with two small configurations. At first, we configured Karma to run tests of an UI5 application. At second, we prepared Azure-Pipelines to visualize the test results. The result showed a test summary in the Azure-Pipeline execution. This functionality helps us to continuously monitor automated test execution and is therefore key of an essential part of a DevOps process.
About the author
Michael Christa is SAP Consultant at Q_PERIOR focusing on technology and innovation. His professional interests are developing end-to-end SAPUI5 applications as well as working with the SAP Business Technology Platform.
- SAP Managed Tags:
