You probably agree that every code you create has to be tested, including your custom code within SAP BW or HANA systems. Especially, if you want to work in an agile mode using automation to solve this challenge is key. Too often, SAP BW developers only use the graphical user interface (GUI) as the preferred testing approach. On the one hand because it is simple to use and on the other side because there were hardly any alternative approaches available. Unfortunately, GUI tests are always End 2 End (E2E) tests and never lead to success alone, but in inadequate runtimes and insufficient coverage. So how to solve this issue? Well, the only feasible approach to solve this is to use the right test strategy. In this article, I explain the reasons why that is, which additional testing methods should be used and what improved testing strategy results out of it.
One thing is clear, there is no Swiss Army knife in software technology or in testing, or a solution that serves everything. Neither can you succeed with GUI tests that always test the entire software stack End2 End. There are several reasons for this, but let’s list the biggest ones:
This test method is probably so much used, because there is a fancy feature: screen recording, which accelerates the speed of test case creation and seems to be quite intuitive in its usage. But one day you will reach the point where solely GUI testing becomes just too expensive, too slow and inefficient. Despite immense efforts, the tests do not run overnight or weekend and you cannot guarantee the quality and delivery dates anymore. Because of that, GUI tests are part of the integration testing and serve, among other things:
For testing the requirements on the smallest granularity (aka Developer Tests), this test method is nowhere near enough, which is why good developers use another one. Developers know that every code change requires testing, once before and after the change. This ensures that everything worked well before the change and that the change did not break any existing feature. So how could GUI testing be used to test just one dedicated object, with no dependency towards other parallel testers, fast and already on the development system? Exactly, it won’t help. That’s why in practice, especially with later changes or hot fixes, testing is not sufficiently done anymore which results in more and more quality issues over time.
At the end this strategy leads into the Ice Cream Anti-Pattern, in which the expenses are distributed the wrong way around:
At SAP Tech-Ed I was often asked what can be done in such a case, if you have reached exactly these limits of GUI testing. My answer to that: flip the efforts and use more Automated Component Tests and less GUI based integration / E2E tests. It were these customers who got to the point to recognize that they tried to solve it all using a single tool.
The solution to the above explained issues is called Component Testing (aka Automates Unit Testing). Within Component Testing an object is isolated in a way that it can be tested independent of the system environment or any other dependent object (e.g.: Dataprovider, Database Table, Calculation View, Function Module, etc.). This is nothing new, many industries work according to this scheme. A car will not be tested on the street for the first time and a laptop will not run the first time after its final assembling. This final “big-bang” integration test an innumerable series of Component Tests are executed in precedence, in which each element is individually isolated and independently validated from the rest. Only if all components have successfully completed all Component Tests, the actual integration tests are performed as an interaction of all (sub) components.
In software Component Testing, parts of the software are tested separately from the rest. It focuses entirely on the validation of the functionality of an ABAP class, SAP BW Transformation, Query, HANA Calculation View, Procedure, etc. This is not simple to do for SAP developers, but the benefits are immense:
Component tests are per se regression tests and fully automated — this is why they are called: Automated Unit Tests (AUT). But when AUTs bring so much, why do many SAP BW or HANA developers barely or never use them? There are a few reasons for this, but the most relevant one is probably a lack of support for SAP BW / HANA. Although there are ABAP Unit Tests and nowadays HANA automated test available, they cannot be used without expert knowledge, are difficult to set up for Transformations or not even available for BEx queries or just not isolated as for SAP HANA tests. In practice it was very beneficial to integrate the business into the process of testing, so they could understand and contribute to test cases. This creates many new benefits, such as the elimination of shadow testing and a high level of confidence in quality assurance on business side. Finger pointing will not exist anymore. But the most important aspect is that a completely different test strategy is feasible if you can create automated (isolated) component tests for all SAP BW / HANA objects used. With xgile.com a tool exists that helps you to create component tests easily for all SAP BW / HANA Objects to go for a new test strategy.
The optimal test strategy is based on the test pyramid (first introduced by Mike Cohen), which states that the tools and efforts should be used according the best cost / benefit ratio. For proving functional correctness there are basically: AUTs, integration tests, and manual tests. Most of the functionality, at 70–80%, should be tested via AUTs, the rest through integration testing, user acceptance tests and manual testing (for example what is too difficult or costly to automate can remain as a manual test).
By shifting the efforts (aka "shift left"), the initial problem, why tests in the SAP BW environment break any deadline and exceed the budget, is resolved by limiting GUI integration tests to its purpose only. Fully automated coverage of all test cases can be achieved, which is the foundation for Dev-Ops and Scrum.
A Scrum cycle without a high test automation, will end up for a SAP BW team in testing only, or no more testing, or only limited testing — probably the most likely scenario.
This also has positive effects on working in the SAP landscape. Most of the testing is already done in the development system, without the need for transportation. This additionally accelerates the development cycles. Errors from the production can easily be simulated and fixed in the development system via AUTs, so they are automated and checked forever.
Due to the high granularity of testing, Component Tests can be directly linked to requirements of the business department, for example from JIRA or HP-QC. The exciting thing about it is that a requirement becomes an AUT and the logic is perfectly documented that way, something I would call an “automated documentation”.
Imagine, external consultants or employees leave the project and their developments are fully secured and documented by automated Component Tests. This will make the difference whether you sweat on later changes or just press a button to validate in seconds that all former requirements are yet met.
There is no doubt that automated component tests are a must to succeed in testing your SAP code to which in the end results in applying a new test strategy. So either you go the hard way and do component testing manually, hoping that the rest of the organisation will understand and follow your example or you use xGile.com to create automated (isolated) component tests without programming via a unified user interface for SAP BW Transformations, Queries, HANA Calculation Views and Procedures. Give it a try, its free of charge for several month. If the report quality is periodically bad and the (manual or automatic) test efforts become too high, then you know the reasons for it now.
This article was first published here.
The Limits of GUI Based E2E Testing
One thing is clear, there is no Swiss Army knife in software technology or in testing, or a solution that serves everything. Neither can you succeed with GUI tests that always test the entire software stack End2 End. There are several reasons for this, but let’s list the biggest ones:
- it requires cost intensive test data management
- it is too slow and too expensive (not efficient), because it is more tested and covered in a test case than required
- too many system dependencies, which is why test cases quickly invalidate
- no comprehensive test design is possible (not effective)
- error investigation on the whole stack and real data is very costly and time consuming
- can only be done on to the test /quality system which is time consuming
This test method is probably so much used, because there is a fancy feature: screen recording, which accelerates the speed of test case creation and seems to be quite intuitive in its usage. But one day you will reach the point where solely GUI testing becomes just too expensive, too slow and inefficient. Despite immense efforts, the tests do not run overnight or weekend and you cannot guarantee the quality and delivery dates anymore. Because of that, GUI tests are part of the integration testing and serve, among other things:
- to test the interplay of the interfaces of objects / modules
- validate process flows (e.g. process chains)
- validate run times
For testing the requirements on the smallest granularity (aka Developer Tests), this test method is nowhere near enough, which is why good developers use another one. Developers know that every code change requires testing, once before and after the change. This ensures that everything worked well before the change and that the change did not break any existing feature. So how could GUI testing be used to test just one dedicated object, with no dependency towards other parallel testers, fast and already on the development system? Exactly, it won’t help. That’s why in practice, especially with later changes or hot fixes, testing is not sufficiently done anymore which results in more and more quality issues over time.
At the end this strategy leads into the Ice Cream Anti-Pattern, in which the expenses are distributed the wrong way around:
Almost Everyone Got to This Point
At SAP Tech-Ed I was often asked what can be done in such a case, if you have reached exactly these limits of GUI testing. My answer to that: flip the efforts and use more Automated Component Tests and less GUI based integration / E2E tests. It were these customers who got to the point to recognize that they tried to solve it all using a single tool.
Quality Assurance Requires Component Testing
The solution to the above explained issues is called Component Testing (aka Automates Unit Testing). Within Component Testing an object is isolated in a way that it can be tested independent of the system environment or any other dependent object (e.g.: Dataprovider, Database Table, Calculation View, Function Module, etc.). This is nothing new, many industries work according to this scheme. A car will not be tested on the street for the first time and a laptop will not run the first time after its final assembling. This final “big-bang” integration test an innumerable series of Component Tests are executed in precedence, in which each element is individually isolated and independently validated from the rest. Only if all components have successfully completed all Component Tests, the actual integration tests are performed as an interaction of all (sub) components.
In software Component Testing, parts of the software are tested separately from the rest. It focuses entirely on the validation of the functionality of an ABAP class, SAP BW Transformation, Query, HANA Calculation View, Procedure, etc. This is not simple to do for SAP developers, but the benefits are immense:
- no “test data management” is required
- testing is already done on the development system because AUTs are independent from the system’s data and other objects (which make them regression tests) which results in very stable tests cases and very fast testing (no transport to the quality system is required anymore)
- results are constant and independent of system state / variables which is also important for regression testing
- a minimal number of just a few test data rows are sufficient (efficient and perfect for debugging)
- test design reduces the number of test cases to the required minimum (efficient)
- everyone can test in parallel to others
Component tests are per se regression tests and fully automated — this is why they are called: Automated Unit Tests (AUT). But when AUTs bring so much, why do many SAP BW or HANA developers barely or never use them? There are a few reasons for this, but the most relevant one is probably a lack of support for SAP BW / HANA. Although there are ABAP Unit Tests and nowadays HANA automated test available, they cannot be used without expert knowledge, are difficult to set up for Transformations or not even available for BEx queries or just not isolated as for SAP HANA tests. In practice it was very beneficial to integrate the business into the process of testing, so they could understand and contribute to test cases. This creates many new benefits, such as the elimination of shadow testing and a high level of confidence in quality assurance on business side. Finger pointing will not exist anymore. But the most important aspect is that a completely different test strategy is feasible if you can create automated (isolated) component tests for all SAP BW / HANA objects used. With xgile.com a tool exists that helps you to create component tests easily for all SAP BW / HANA Objects to go for a new test strategy.
The Optimal Test Strategy for SAP BW / HANA
The optimal test strategy is based on the test pyramid (first introduced by Mike Cohen), which states that the tools and efforts should be used according the best cost / benefit ratio. For proving functional correctness there are basically: AUTs, integration tests, and manual tests. Most of the functionality, at 70–80%, should be tested via AUTs, the rest through integration testing, user acceptance tests and manual testing (for example what is too difficult or costly to automate can remain as a manual test).
By shifting the efforts (aka "shift left"), the initial problem, why tests in the SAP BW environment break any deadline and exceed the budget, is resolved by limiting GUI integration tests to its purpose only. Fully automated coverage of all test cases can be achieved, which is the foundation for Dev-Ops and Scrum.
A Scrum cycle without a high test automation, will end up for a SAP BW team in testing only, or no more testing, or only limited testing — probably the most likely scenario.
This also has positive effects on working in the SAP landscape. Most of the testing is already done in the development system, without the need for transportation. This additionally accelerates the development cycles. Errors from the production can easily be simulated and fixed in the development system via AUTs, so they are automated and checked forever.
Due to the high granularity of testing, Component Tests can be directly linked to requirements of the business department, for example from JIRA or HP-QC. The exciting thing about it is that a requirement becomes an AUT and the logic is perfectly documented that way, something I would call an “automated documentation”.
Imagine, external consultants or employees leave the project and their developments are fully secured and documented by automated Component Tests. This will make the difference whether you sweat on later changes or just press a button to validate in seconds that all former requirements are yet met.
Conclusion
There is no doubt that automated component tests are a must to succeed in testing your SAP code to which in the end results in applying a new test strategy. So either you go the hard way and do component testing manually, hoping that the rest of the organisation will understand and follow your example or you use xGile.com to create automated (isolated) component tests without programming via a unified user interface for SAP BW Transformations, Queries, HANA Calculation Views and Procedures. Give it a try, its free of charge for several month. If the report quality is periodically bad and the (manual or automatic) test efforts become too high, then you know the reasons for it now.
This article was first published here.
- SAP Managed Tags:
