An effective test suite, which is fast, maintainable and robust is critical for sustainable productivity, confidence in adapting the software to changing needs and high quality - especially when moving to cloud development and continuous delivery. Test Automation should be able to quickly catch bugs, and this makes it possible to change software with confidence, which enables teams to iterate faster and adapt more quickly to changing market needs, technologies or customer requirements. Writing automated tests early also improves the design of your system - it forces teams to confront architecture and design issues from the beginning, which leads to more modular and adaptable software.
In the industry there can sometimes be some confusion on how different testing terms are used and understood. To improve communication, reflection and learning within SAP - a larger part of the engineering community agreed on one terminology, which is used then for trainings on test automation, documentation of good practices, giving additional knowledge sharing sessions or reflecting about necessary architecture improvements. We apply the same terminology to different technology stacks (e.g. ABAP, Java, JavaScript ...).
Since I get more and more requests from larger customers and partners on what kind of test strategies are implement by SAP teams, I want to publish some blogs during the coming years on test automation and test strategy. This blog focuses on the terminology for the test pyramid - which is one of the many models to think and communicate about test strategy. The blog assumes that the reader already has an understanding of test isolation, fakes, dependency injection and test doubles. If you that is not the case I strongly recommend to read more about it on Gerard Meszaros xUnit Patterns.
The Test Pyramid gives advice on where to test what, by which kind of test and where to invest how much effort. The essential point of the test pyramid is that you should have more low-level tests (e.g. unit or component tests) than high level system tests. Low-level tests are much faster than high-level tests, because of the smaller scope and higher isolation from dependencies and they give us excellent defect localization (show cause of a failure) and avoid fragility causing a lot of false positives when running the tests. Due to these factors they also allow to test a much higher variance and you can create new tests very quickly.
The following graphic shows a visualization of the test pyramid and afterwards the corresponding terms are explained.
Unit tests focus on a single unit, which can be a class, method or sometimes even a bunch of classes together. Slow, fragile, uncontrollable, unobservable or costly dependencies are replaced by test doubles.
Examples for test tools are JUnit, ABAPUnit, QUnit or Jest.
Component tests check whether a component (e.g. set of classes or modules) works. Only dependencies outside of the component are replaced by test doubles.
Examples for test tools are JUnit, ABAPUnit, QUnit or Jest.
Integration tests check whether the interaction between some components work. Slow, fragile, uncontrollable, unobservable and costly dependencies are replaced by test doubles, but the checked components are used in its productive version.
Examples for test tools are JUnit, ABAPUnit, QUnit or Jest. Combined with specialized utilities. A typical example for such utilities on this layer, are Spring Boot Integration Tests or OPA5 tests for testing of UI5 applications in isolation from the backend.
System tests check the whole system in a black-box manner. In most cases nothing is doubled, everything is the productive implementation. Still there can be the need to fake things. E.g. the test data that may be created in the database directly, or actions that might be triggered by double, such as asynchronous messages (e.g. simulated RabbitMQ messages).
System Tests can test against the User Interface or Services. Examples for test tools against the UI are webdriver.io, cypress.io or puppeteer. Examples for test tools against the service / API are REST Assured, Karate or supertest.
Scenario tests check a set of whole systems in a black-box manner. Nothing is doubled, everything is real. Still there can be the need to fake things. E.g. the test data that may be created in the database directly, or actions that might be triggered by double, such as asynchronous messages (e.g. simulated RabbitMQ messages).
The test tools that are used to implement the tests, such as ABAP Unit, JUnit, QUnit, Jasmine, OPA, Selenium, Cucumber, etc. can be used for different levels of the test pyramid. Although most technologies were invented with a specific scope in mind (e.g. unit tests frameworks) - they can be used for different scopes as well. For example, although JUnit or ABAP Unit were invented for unit tests, you can easily use it to write component, integration and system tests as well.
The blog does not make any recommendation regarding test tools for the different levels, the named tools are just examples to make the terms clearer. Tool recommendations for different SAP technology stacks could be a topic for future blog posts. As well as a test strategy perspective on shift left and continuous delivery. So stay tuned.
If you do not miss an update on Clean ABAP, Clean SAPUI5, test automation, testability and other engineering / craftsmanship / architecture topics, subscribe to the brand new newsletter. The newsletter will not only be used for sharing knowledge, but also offer opportunities for collaboration, building communities and co-creation.
In the industry there can sometimes be some confusion on how different testing terms are used and understood. To improve communication, reflection and learning within SAP - a larger part of the engineering community agreed on one terminology, which is used then for trainings on test automation, documentation of good practices, giving additional knowledge sharing sessions or reflecting about necessary architecture improvements. We apply the same terminology to different technology stacks (e.g. ABAP, Java, JavaScript ...).
Since I get more and more requests from larger customers and partners on what kind of test strategies are implement by SAP teams, I want to publish some blogs during the coming years on test automation and test strategy. This blog focuses on the terminology for the test pyramid - which is one of the many models to think and communicate about test strategy. The blog assumes that the reader already has an understanding of test isolation, fakes, dependency injection and test doubles. If you that is not the case I strongly recommend to read more about it on Gerard Meszaros xUnit Patterns.
The Test Pyramid gives advice on where to test what, by which kind of test and where to invest how much effort. The essential point of the test pyramid is that you should have more low-level tests (e.g. unit or component tests) than high level system tests. Low-level tests are much faster than high-level tests, because of the smaller scope and higher isolation from dependencies and they give us excellent defect localization (show cause of a failure) and avoid fragility causing a lot of false positives when running the tests. Due to these factors they also allow to test a much higher variance and you can create new tests very quickly.
The following graphic shows a visualization of the test pyramid and afterwards the corresponding terms are explained.
Test Pyramid
Unit Tests
Test against a single unit and isolated from the environment
Unit tests focus on a single unit, which can be a class, method or sometimes even a bunch of classes together. Slow, fragile, uncontrollable, unobservable or costly dependencies are replaced by test doubles.
Examples for test tools are JUnit, ABAPUnit, QUnit or Jest.
Component Tests
Test against one component and isolated from other components
Component tests check whether a component (e.g. set of classes or modules) works. Only dependencies outside of the component are replaced by test doubles.
Examples for test tools are JUnit, ABAPUnit, QUnit or Jest.
Integration Tests
Test the integration of some components and isolated from other components
Integration tests check whether the interaction between some components work. Slow, fragile, uncontrollable, unobservable and costly dependencies are replaced by test doubles, but the checked components are used in its productive version.
Examples for test tools are JUnit, ABAPUnit, QUnit or Jest. Combined with specialized utilities. A typical example for such utilities on this layer, are Spring Boot Integration Tests or OPA5 tests for testing of UI5 applications in isolation from the backend.
System Tests (UI and Service)
Test the complete stack through UI or exposed service
System tests check the whole system in a black-box manner. In most cases nothing is doubled, everything is the productive implementation. Still there can be the need to fake things. E.g. the test data that may be created in the database directly, or actions that might be triggered by double, such as asynchronous messages (e.g. simulated RabbitMQ messages).
System Tests can test against the User Interface or Services. Examples for test tools against the UI are webdriver.io, cypress.io or puppeteer. Examples for test tools against the service / API are REST Assured, Karate or supertest.
Scenario Tests
System test including many steps across different applications or services
Scenario tests check a set of whole systems in a black-box manner. Nothing is doubled, everything is real. Still there can be the need to fake things. E.g. the test data that may be created in the database directly, or actions that might be triggered by double, such as asynchronous messages (e.g. simulated RabbitMQ messages).
Test Tools not tight to a specific level
The test tools that are used to implement the tests, such as ABAP Unit, JUnit, QUnit, Jasmine, OPA, Selenium, Cucumber, etc. can be used for different levels of the test pyramid. Although most technologies were invented with a specific scope in mind (e.g. unit tests frameworks) - they can be used for different scopes as well. For example, although JUnit or ABAP Unit were invented for unit tests, you can easily use it to write component, integration and system tests as well.
The blog does not make any recommendation regarding test tools for the different levels, the named tools are just examples to make the terms clearer. Tool recommendations for different SAP technology stacks could be a topic for future blog posts. As well as a test strategy perspective on shift left and continuous delivery. So stay tuned.
Related Resources
- Testable Code: If you want to learn more on how to write testable ABAP code. There is a whole section in Clean ABAP, Clean ABAP book and a free training on OpenSAP Write Testable ABAP Code.
- Testable UI5 Code: There is a tutorial, which also touches on how to [write testable UI5 JavaScript code}(https://openui5.hana.ondemand.com/topic/291c9121e6044ab381e0b51716f97f52).
Subscribe to Newsletter: Collaboration on Improving
If you do not miss an update on Clean ABAP, Clean SAPUI5, test automation, testability and other engineering / craftsmanship / architecture topics, subscribe to the brand new newsletter. The newsletter will not only be used for sharing knowledge, but also offer opportunities for collaboration, building communities and co-creation.
- SAP Managed Tags:
1 Comment
