Test management. Automated tools comparison
In this article I will try to compare provided tools for SAP applications testing. This analisys were made for one of SAP customers.
There are three known options for SAP products testing:
- Using SAP Solution Manager Test Workbench
- Using HP tools (SAP Quality Center by HP)
- Using IBM Rational tools
At the same time, much attention is paid to testing automation which helps to solve such problems as:
- A large number of the tested business processes
- High labor input and cost of manual testing
- A human factor when testing
- Availability of necessary human resources for carrying out testing
- Complexity of scoping of testing
- Increase in complexity of testing in case of increase in number of business processes
However in the automated testing not everything is so simple.
High requirements of qualification of employees during creation of test scripts are imposed. It is also rather difficult to support the existing scripts in an actual condition in case of frequent change of business processes.
For the solution of similar problems there was developed so-called “Component-Based testing”
Component-Based testing of business processes allows to simplify creation and support of the automated test scripts for the users who are responsible for testing.
- Instead of working directly with a script, the tester can build the scenario, using ready components of the test, each of them represents the written-down script.
- The instrument of component testing scans the application and generates the automated script consisting of a set of components of the test.
- This script can be used further for testing. Testers can independently carry out such scanning since knowledge of programming of scripts isn’t required.
- The generated scenario can be united in an independent business component which can be used further for creation of other scripts
Time required for manual testing are the greatest for an overall picture of testing. However it is the most economic option for the scenarios tested once or very rare. Despite a possibility of reuse of the written-down automated scenarios, costs for their support are big since scripts improve manually. The smallest labor costs on support are required for component testing since there is a possibility of repeated scanning of the tested process without their rewriting.
The following evaluation criteria have been developed for further assessment, taking into account the aforesaid
Functional capabilities
|
Name of evaluation criterion |
Description of evaluation criterion |
|
Test Plans generation |
Possibility of drawing up plans of testing on the basis of the generated scripts. Possibility of use by the subsequent scripts of results previous. |
|
Existence of the tool for record of a test script |
Existence of mechanisms for recording of the user actions made in system with a possibility to use this record at the subsequent testing. |
|
Existence of additional tools of component testing for optimization of works on generation and support of a test repository |
Availability of additional instruments of record of the user actions facilitating initial generation of scripts with a possibility of their automated updating in case of change of the corresponding business process, for example SAP TAO |
|
Possibility of the automated testing of ABAP developments |
Support of test automation of ABAP developments |
|
Possibility of the automated testing of Java developments |
Support of automation of Java developments for the SAP applications |
|
Possibility of the automated testing of developments of SAP Webdynpro for Java |
Support of automation of testing of developments of Webdynpro for Java |
|
Возможность автоматизированного тестирования разработок SAP Webdynpro for ABAP |
Possibility of the automated testing of developments of SAP Webdynpro for ABAP |
|
Possibility of the automated testing of developments of SAP CRM Web UI |
Support of automation of testing of developments of Web UI |
|
Analysis and Reporting |
Possibility of generation of reports on testing and simplicity of their generation |
|
Support of regular load testing of SAP systems |
Possibility of regular carrying out and comparison of several load tests on the chosen system in a uniform window with comparison of trends on the executed transactions |
|
Possibility of automation of testing of mobile platforms for SAP systems |
Possibility of automation of functional regression testing of mobile applications on various devices |
|
Existence of a possibility of adjustment of loading at load testing |
Adjustment of loading when conducting load testing |
Integration with application lifecycle management processes
|
Name of evaluation criterion |
Description of evaluation criterion |
|
Integration with Project Documentation |
Support of business processes structure. Possibility of transfer of structure of the project from SAP Solution Manager. Updating of structure of the SAP Solution Manager project based on results of accomplishment of scripts. Determination of a covering test scripts of specific elements of structure of business processes |
|
Integration with SAP Solution Manager ITSM scenarios |
Possibility of maintaining errors of testing in Solution Manager. Existence of automatic communication of an incident and test unit |
|
Integration with Change Management of SAP SM |
Possibility of use of the tool for testing of change operations of Solution Manager in a test landscape. Transfer of test results in change operation |
|
Integration with BPCA |
Possibility to use of results of work of SAP Solution Manager Business Process Change Analyser (BPCA) intended for the analysis of changes in system with the subsequent generation and optimization of test plans |
Ease of use
|
Name of evaluation criterion |
Description of evaluation criterion |
|
Simplicity of a script recording |
Efforts on initial record of a script |
|
Simplicity of debugging of a script |
Efforts for debugging of a script (adding of breakpoints, computing logic and etc) |
|
Simplicity of updating of a script |
Efforts for updating of a script in case of change of business process in change transaction |
|
Simplicity of support of input data |
Complexity of definition/updatingof input data of a script |
|
Usability |
Simplicity of the interface and its clearness for the ultimate user |
|
Simplicity and convenience of administration |
Simplicity and convenience of administration |
|
Existence of the built-in means of the organization of a role model |
Existence of model of differentiations of powers in a product. |
On the basis of the criteria stated above, assessment of the offered tools has been carried out. Prioritezation of criteria has been executed by a customer on the basis of his requirements from 1 – not important, to 5 – very important. Opportunity/extent of realization can be expressed or as parameter realizable/unrealizable (1/0) or as extent of realization from 1 to 5
Functional capabilities analysis
|
Name of evaluation criterion |
Priority |
SAP SM |
SAP QC by HP |
IBM Rational |
|
|
Test Plans generation |
4 |
1 |
1 |
1 |
|
|
Existence of the tool for record of a test script |
4 |
1 |
1 |
1 |
|
|
Existence of additional tools of component testing for optimization of works on generation and support of a test repository |
4 |
1 |
1 |
0 |
|
|
Possibility of the automated testing of ABAP developments |
5 |
1 |
1 |
1 |
|
|
Possibility of the automated testing of Java developments |
5 |
1 |
1 |
1 |
|
|
Possibility of the automated testing of developments of SAP Webdynpro for Java |
5 |
1 |
1 |
1 |
|
|
Возможность автоматизированного тестирования разработок SAP Webdynpro for ABAP |
5 |
1 |
1 |
1 |
|
|
Possibility of the automated testing of developments of SAP CRM Web UI |
5 |
1 |
1 |
1 |
|
|
Analysis and Reporting |
5 |
1 |
1 |
1 |
|
|
Support of regular load testing of SAP systems |
5 |
0 |
1* |
1* |
|
|
Possibility of automation of testing of mobile platforms for SAP systems |
5 |
1 |
1 |
0 |
|
|
Existence of a possibility of adjustment of loading at load testing |
5 |
0 |
1 |
1 |
|
Integration with application lifecycle management processes analisys
|
Name of evaluation criterion |
Priority |
SAP SM |
SAP QC by HP |
IBM Rational |
|
Integration with Project Documentation |
4 |
1 |
1 |
1 |
|
Integration with SAP Solution Manager ITSM scenarios |
3 |
1 |
1 |
1 |
|
Integration with Change Management of SAP SM |
3 |
1 |
0 |
0 |
|
Integration with BPCA |
4 |
1 |
1 |
1 |
Ease of use analysis
|
Name of evaluation criterion |
Priority |
SAP SM |
SAP QC by HP |
IBM Rational |
|
Simplicity of a script recording |
5 |
5 |
5 |
5 |
|
Simplicity of debugging of a script |
4 |
5 |
5 |
3 |
|
Simplicity of updating of a script |
3 |
5 |
5 |
3 |
|
Simplicity of support of input data |
4 |
5 |
5 |
4 |
|
Usability |
5 |
5 |
5 |
5 |
|
Simplicity and convenience of administration |
3 |
5 |
5 |
5 |
|
Existence of the built-in means of the organization of a role model |
4 |
5 |
1 |
1 |
|
Tool cost |
5 |
5 |
3 |
4 |
Total:
• On the SAP Solution Manager 226 points
• On the HP 207 points
• On the IBM Rational 185 points
By results of assessment the SAP Solution Manager tools and HP tools are approximately identical by opportunities. Besides it is possible to carry to pluses of HP that this tool is long enough used in the market and is suitable for testing not only the SAP applications. But the SAP Solution Manager tools are free in the presence for Enterprise Support clients.
Further we will in detail consider the offered tools