This is the fourth part of the blog-series about automated testing (end to end and integration-tests) using testcafe. The blog-series has the following parts:
Inside this 4th part we will implement code-coverage for e2e tests and realize integration tests using testcafe.
Code Coverage can be used to identify the scope of your tests. The results are showing you which line of code, which code-branch (If/else/..) and which procedures/functions were called by your tests. Code coverage is especially used within Unit-Tests, to measure the quality of your tests - some projects are giving minimal code-coverage-requirements (e.g. 80% of all lines) to ensure quality. To implement code coverage your code is normally instrumented before the test-execution. High-Level technically speaking for every line, method and branch of your code, additional code is added by the "instrument-tool", which is recognizing the call. Imagine the following method:
after instrumentation the code will look (pseudo-code) more or less like that:
For E2E tests code-coverage is pretty uncommon, as your code is deployed and productive. Instrumenting such code, can be very difficult, if you don't want to deploy an instrumented version of your code (which you shouldn't.. as you can imagine, this code is much slower). Additionally e2e tests should have the goal to smoke test the application - not to ensure that every single edge case is tested.
Therefore as a word of warning: Code-Coverage is implemented by the ui5 enhancements for testcafe - but the execution will be much slower, and therefore I would not recommend to have it activated for productive test execution (e.g. on a Jenkins pipeline). Goal of code-coverage for E2E should be to easily identify major areas of functionality (by code) which your test are not covering. Do not try to reach a fixed percentage of code-coverage for your E2E test code. Generally: Less Stable and high-quality tests >> Low-Quality tests with high code-coverage.
To implement code-coverage for E2E tests, the ui5 enhancements for testcafe is creating a proxy-server before starting your test-cases. The proxy is using the framework "istanbul" to instrument your application code and to generate the coverage output. The sample code is (as always) available on github . Just clone the repo and run it from there, if you are having any issues following the steps in this blog post. In this blog post we are using the same example test as within blog-post 3.
To enable the code-coverage functionality, we have to enhance the ".ui5-testcafe.json" file.
Let's go over all settings:
After maintaing those settings, the coverage is recorded and merged over multiple testcafe tests - still nothing is reported, as the last step is missing: We have to install the testcafe reporter ui5-coverage to enable the output of the coverage results. Execute the instrution:
to install the plugin for testcafe. Afterwards enhance the reporter section in the .testcaferc.json file, to enable the reporter.
If you run your test now, you will notice that the output is containg logs about what is instrumentd and what not. As an example have a look at the following gif:
At the end of the test the coverage output is added to the console output.
Next to the summary a full coverage report is generated. Just open the file and you will get the full coverage report of your test.
When clicking on the files you can directly check the coverage of the selected files on line/branch level.
Until now we only talked about E2E tests. So tests which are running against an actual backend with real data, and a deployed/productive application hosted e.g. on a FIORI Launchpad. For integration tests normally OPA5 is used. I personally like the overall syntax/coding style of testcafe much more - therefore I also prefer to use it for integration tests. The code for this example is available in GitHub. The following part of the blog is assuming that you are already using the ui5-tooling. If you are not using it - use it. It is absolutly worth it, and much better than any other currently available solution.
The following chapter is also based on the Shopping-Cart demo. Please download the shopping-cart demo application from the sapui5 documentation.(Download-Button). Afterwards remove the existing content from the tests folder, and copy the pages and test files from the example before.
To enable testcafe against the local developments, we have to start the ui5-server and run the test against the localhost server. This is a small problem, as testcafe shouldn't start before the server is up and running - which can take just a few milliseconds, but also a few seconds. In theory testcafe has a CLI Parameter (--app) to start any command before testcafe starts. This is unfortunatly not very stable. Therefore we will use the two npm packages concurrently and wait-on. Concurrently allows to execute multiple commands (Server Start && testcafe) in paralell. Wait-On allows to pause a command (testcafe) until a specific condition (host and port available) is met. Too make our life easier and not always enter a long CLI instruction, we will maintain the startup script inside the scripts section in the package.json.
Again a complex instruction - let's have a look in depth:
Additionally we have to adjust our test-fixture to access the localhost port provided by ui5-server.
That's it already. Start the test by running "npm run int-test". Of course you can also access other files than index.html. If you have (e.g.) a seperate index file for your application running on a mockserver, you can of course also specify this file.
What is missing now of course is the code-coverage for integration tests. For integration tests code coverage is much more important, as integration tests (e.g. running against a mock-server) should already catch most of your code and even a few edge cases. For Code-Coverage the existing proxy server by ui5-server (ui5-tooling) is enhanced using the npm package ui5-middleware-testcafe-coverage, which is instrumenting your files on the fly from the webapp folder. The same settings as within the code-coverage for e2e tests are respected. The only difference is, that the "proxy" parameter must be set on false.
Summed up the following steps are required:
2. Add the custom middleware to your ui5.yaml (don't worry. the middleware does not have any effect in case coverage is not enabled).
3. Add the coverage configuration to your ui5-testcafe.json file (note that we are not setting enabled on true, as we don't want the coverage to be activated globally. instead we will activate the coverage collection via enviroment variable UI5_COVERAGE_ENABLED ).
4. Add the coverage reporter to your testcafe configuration file
5. Add a new run-script, which is doing exactly the same like in the previous chapter, but also sets the enviroment variable UI5_COVERAGE_ENABLED
That's it. If you run the script int-test-cov via "npm run int-test-cov", the output will contain coverage information (by accident, exactly the same numbers like before 🙂 )
Using this blog post you should be able to fully use testcafe inside your integration tests (as a replacement for OPA5).
- Part 1: Overview and first test
- Part 2: Enhanced Selection, Action and Assertion Possibilities.
- Part 3: Structure your test and common best-practices
- Part 4: (this blog) Code-Coverage E2E tests, Integration Tests
- Part 5: Integration in CI (Jenkins / Github Actions) / Authentification
- Part 6: Use testcafe for SAP Analytics Cloud and SAP Lumira
Inside this 4th part we will implement code-coverage for e2e tests and realize integration tests using testcafe.
UI Code Coverage for E2E tests
Code Coverage can be used to identify the scope of your tests. The results are showing you which line of code, which code-branch (If/else/..) and which procedures/functions were called by your tests. Code coverage is especially used within Unit-Tests, to measure the quality of your tests - some projects are giving minimal code-coverage-requirements (e.g. 80% of all lines) to ensure quality. To implement code coverage your code is normally instrumented before the test-execution. High-Level technically speaking for every line, method and branch of your code, additional code is added by the "instrument-tool", which is recognizing the call. Imagine the following method:
function test() {
document.title = "test";
}after instrumentation the code will look (pseudo-code) more or less like that:
function test() {
window.__coverage__["function_test"].called ++;
window.__coverage__["line_2"].called ++;
document.title = "test";
}For E2E tests code-coverage is pretty uncommon, as your code is deployed and productive. Instrumenting such code, can be very difficult, if you don't want to deploy an instrumented version of your code (which you shouldn't.. as you can imagine, this code is much slower). Additionally e2e tests should have the goal to smoke test the application - not to ensure that every single edge case is tested.
Therefore as a word of warning: Code-Coverage is implemented by the ui5 enhancements for testcafe - but the execution will be much slower, and therefore I would not recommend to have it activated for productive test execution (e.g. on a Jenkins pipeline). Goal of code-coverage for E2E should be to easily identify major areas of functionality (by code) which your test are not covering. Do not try to reach a fixed percentage of code-coverage for your E2E test code. Generally: Less Stable and high-quality tests >> Low-Quality tests with high code-coverage.
To implement code-coverage for E2E tests, the ui5 enhancements for testcafe is creating a proxy-server before starting your test-cases. The proxy is using the framework "istanbul" to instrument your application code and to generate the coverage output. The sample code is (as always) available on github . Just clone the repo and run it from there, if you are having any issues following the steps in this blog post. In this blog post we are using the same example test as within blog-post 3.
To enable the code-coverage functionality, we have to enhance the ".ui5-testcafe.json" file.
{
"coverage": {
"enabled": true,
"log": true,
"proxy": true,
"basePath": "/test-resources/sap/m/demokit/cart/webapp",
"type": "html",
"outDir": "./report/coverage"
"debugComponents": [
"sap/ui/demo/cart"
],
"includePaths": [],
"excludePaths": []
},
//..
}Let's go over all settings:
- enabled: Core setting, which enables the code-coverage. You can also use the enviroment variable UI5_COVERAGE_ENABLED to globally deactivate / activate code-coverage.
- log: If activated during the test run you will get an output of all instrumented and skipped files.
- proxy: Activates the proxy described above. You have to activate the proxy in case your code is not already instrumented.
- basePath: You are normally only interested in specific applications and don't want to instrument every single application (e.g. avaialble on a Fiori Launchpad). Especially you should avoid to instrument sapui5 resources, and this will make your test horribly slow, and does not really bring any benefit (besides maybe a good feeling to access 10% of UI5 inside your code? 🙂 ). Therefore you should provide a basePath. All files which are inside this basePath are instrumented.
- type: Output-Type of the coverage-report. HTML is the default. The setting is forewarded to istanbul.
- outDir: folder to which the coverage report is written. Can be relative and absolute.
- debugComponents: Of course instrumentation and code-coverage only makes sense for non-minified / uglified code. By default (if you are using ui5 tooling) your productive code should be minified of course. Therefore the instrumenter is skipping all "library-preload", and "Component-preload" files. To enable code coverage of your application your should provide all components you want to cover inside this array. These components will be added to the "sap-ui-debug" URL parameter, causing UI5 to load the "-dbg" version of your code.
- includePaths: Optional list of relative paths you want to whitelist inside your basePath
- excludePaths: Optional list of relative paths you want to blacklist inside your basePath
After maintaing those settings, the coverage is recorded and merged over multiple testcafe tests - still nothing is reported, as the last step is missing: We have to install the testcafe reporter ui5-coverage to enable the output of the coverage results. Execute the instrution:
npm i testcafe-reporter-ui5-coverageto install the plugin for testcafe. Afterwards enhance the reporter section in the .testcaferc.json file, to enable the reporter.
//..
"reporter": [
{
"name": "spec"
},
{
"name": "ui5-coverage"
}
]
//..If you run your test now, you will notice that the output is containg logs about what is instrumentd and what not. As an example have a look at the following gif:
At the end of the test the coverage output is added to the console output.
Next to the summary a full coverage report is generated. Just open the file and you will get the full coverage report of your test.
When clicking on the files you can directly check the coverage of the selected files on line/branch level.
Integration Tests
Until now we only talked about E2E tests. So tests which are running against an actual backend with real data, and a deployed/productive application hosted e.g. on a FIORI Launchpad. For integration tests normally OPA5 is used. I personally like the overall syntax/coding style of testcafe much more - therefore I also prefer to use it for integration tests. The code for this example is available in GitHub. The following part of the blog is assuming that you are already using the ui5-tooling. If you are not using it - use it. It is absolutly worth it, and much better than any other currently available solution.
The following chapter is also based on the Shopping-Cart demo. Please download the shopping-cart demo application from the sapui5 documentation.(Download-Button). Afterwards remove the existing content from the tests folder, and copy the pages and test files from the example before.
To enable testcafe against the local developments, we have to start the ui5-server and run the test against the localhost server. This is a small problem, as testcafe shouldn't start before the server is up and running - which can take just a few milliseconds, but also a few seconds. In theory testcafe has a CLI Parameter (--app) to start any command before testcafe starts. This is unfortunatly not very stable. Therefore we will use the two npm packages concurrently and wait-on. Concurrently allows to execute multiple commands (Server Start && testcafe) in paralell. Wait-On allows to pause a command (testcafe) until a specific condition (host and port available) is met. Too make our life easier and not always enter a long CLI instruction, we will maintain the startup script inside the scripts section in the package.json.
{
"name": "ShoppingCart",
//...
"scripts": {
"int-test": "concurrently --success first --kill-others \"ui5 serve --h2\" \"wait-on https://localhost:8443&&testcafe ./webapp/test/testcafe/**\"",
},
//...
}Again a complex instruction - let's have a look in depth:
- concurrently: Allows to start multiple processes at once
- --kill others: Instructs concurrently to kill the second process as soon as the first process stops. This is required to shut down the ui5-server when testcafe is done.
- --success first: Instructs concurrently to stop the process with the exit code of the first finished process. If testcafe finishs with success, the overall process will finish with success
- ui5 serve --h2: first process. Starts the ui5-server
- wait-on https.//localhost:8443: Waits for localhost port 8443 to be available
- &&: Sequential operator. The next instruction (Testcafe) will be executed as soon as the one before is done.
testcafe ./webapp/test/testcafe/** Starts testcafe with all test files inside webapp/test/testcafe.
Additionally we have to adjust our test-fixture to access the localhost port provided by ui5-server.
//....
ui5Fixture('Startseite', "SAP", "https://localhost:8443/index.html");
ui5Test('Create and Order Shopping-Cart', async u => {
//...
});That's it already. Start the test by running "npm run int-test". Of course you can also access other files than index.html. If you have (e.g.) a seperate index file for your application running on a mockserver, you can of course also specify this file.
Integration Tests and Code-Coverage
What is missing now of course is the code-coverage for integration tests. For integration tests code coverage is much more important, as integration tests (e.g. running against a mock-server) should already catch most of your code and even a few edge cases. For Code-Coverage the existing proxy server by ui5-server (ui5-tooling) is enhanced using the npm package ui5-middleware-testcafe-coverage, which is instrumenting your files on the fly from the webapp folder. The same settings as within the code-coverage for e2e tests are respected. The only difference is, that the "proxy" parameter must be set on false.
Summed up the following steps are required:
- install ui5-middleware-testcafe-coverage
npm i ui5-middleware-testcafe-coverage --save-dev2. Add the custom middleware to your ui5.yaml (don't worry. the middleware does not have any effect in case coverage is not enabled).
server:
customMiddleware:
- name: ui5-middleware-testcafe-coverage
afterMiddleware: compression
mountPath: /3. Add the coverage configuration to your ui5-testcafe.json file (note that we are not setting enabled on true, as we don't want the coverage to be activated globally. instead we will activate the coverage collection via enviroment variable UI5_COVERAGE_ENABLED ).
{
"coverage": {
"log": false,
"proxy": false,
"basePath": "",
"type": "html",
"debugComponents": [],
"includePaths": [],
"excludePaths": [
"/resources" //contains sap resources we are normally not interested in
]
}
}4. Add the coverage reporter to your testcafe configuration file
"reporter": [
{
"name": "spec"
},
{
"name": "ui5-coverage"
}
]5. Add a new run-script, which is doing exactly the same like in the previous chapter, but also sets the enviroment variable UI5_COVERAGE_ENABLED
"scripts": {
"int-test": "concurrently --success first --kill-others \"ui5 serve --h2\" \"wait-on https://localhost:8443&&testcafe ./webapp/test/testcafe/**\"",
"int-test-cov": "concurrently --success first --kill-others \"SET UI5_COVERAGE_ENABLED=true&&ui5 serve --h2 \" \"SET UI5_COVERAGE_ENABLED=true&&wait-on https://localhost:8443&&testcafe ./webapp/test/testcafe/**\""
},That's it. If you run the script int-test-cov via "npm run int-test-cov", the output will contain coverage information (by accident, exactly the same numbers like before 🙂 )
Using this blog post you should be able to fully use testcafe inside your integration tests (as a replacement for OPA5).
- SAP Managed Tags:
