How fascinating it is to find, if we can measure the Quality of a product by just looking at few Quality Metrics. I am sure, it will grab attention of many who are really into Quality matters.
Quality metrics can be designed in many dimensions (Here to list a few - usability /performance /functionality /Accessibility /localization /Security etc..). I am here talking about coverage metrics for a product. Product Test Coverage can be achieved & measured considering three important aspects or metrics – Functional coverage, Code coverage and % Execution; to achieve & measure the final inherent Quality of a Product.
Before we decide which one is more important, let’s understand what each of these metrics means in Quality world:
Functional Coverage: This quality metric reflects the total functionalities which are tested, how many of them are working and meeting the acceptance criteria. This is very important metric for customer satisfaction or moreover, for meeting the real definition of what is “Quality” of product.
Code coverage: This quality metric is an indicator of total no of code lines/statements exercised in the test suite, internally hitting development packages or application components.
% Execution: This metric is best to describe, when quality organization stakeholders want to know how much % of test suite have been covered. Typically, 90-95% test coverage is a very good indicator for any test phase.
Functional coverage is the first & foremost requirement to test a product i.e. product should conform to the requirements it is designed for, with specifying the limitations, if there are any. Additionally, testing using different variants – testing boundary conditions, decision tables, branches etc.; ensures that product is doing well in all conditions, for desired functionalities. On the other hand, while how many code lines (statements) are getting exercised during the test, indicates Code coverage, this alone is not a healthy quality metrics, because a lot of code statement (e.g. Function Module) can be covered by just executing it with some random or unrelated junk values. Here though code lines will still be traversed and the code coverage metric will appear good, the random data chosen for testing will not reflect conformance to the functionality requirements.
Looking at any of above metric alone can give us an incomplete picture of quality of a product. Hence it is important to take both Functional & Code coverage in consideration, while defining the quality metrics or evaluating a product’s quality. These two metrics are further complemented by % Execution which ensures that a minimum number of test (~ 95%) have been executed.
On that note, let me also cover briefly about an important SAP tool which helps in analyzing the test coverage - Coverage Analyzer SCOV. It’s a very powerful system-wide runtime analysis tool for monitoring and analyzing the code covered by test suites such as Automated tests, Manual Tests, Unit tests, integration tests.
By making use of SCOV coverage results for (Automated /Manual test suite), we can enhance our functional tests in such a way that we cover the untouched statements/paths/messages in code to enhance the quality of our tests and cover more. Using this tool, we are not only targeting enhanced Functional coverage but complementing it with enhanced code coverage too.
Here is an example to enhance the Code coverage by introducing different test conditions:
Initial Code snippet:
First Test condition: Don’t pass the parameters /pass wrong parameters to retrieve the results to traverse the statement NO_RECORD_FOUND.
Test Condition 2: Don’t pass the mandatory parameters completely to traverse PARAMETER_INCOMPELETE condition.
I have spent my 12 years in Quality management, testing SAP products (Industry solutions) in various releases and have experienced that considering these three aspects(metrics) of coverage are very important in defining & evaluating quality of product. It really helps achieving overall higher quality, higher customer satisfaction, higher reliability and lesser rework.
We at Suite Test Engineering, are leveraging this new test approach to ensure "High Quality" product delivery to our end-users and customers.
Quality metrics can be designed in many dimensions (Here to list a few - usability /performance /functionality /Accessibility /localization /Security etc..). I am here talking about coverage metrics for a product. Product Test Coverage can be achieved & measured considering three important aspects or metrics – Functional coverage, Code coverage and % Execution; to achieve & measure the final inherent Quality of a Product.
Before we decide which one is more important, let’s understand what each of these metrics means in Quality world:
Functional Coverage: This quality metric reflects the total functionalities which are tested, how many of them are working and meeting the acceptance criteria. This is very important metric for customer satisfaction or moreover, for meeting the real definition of what is “Quality” of product.
Code coverage: This quality metric is an indicator of total no of code lines/statements exercised in the test suite, internally hitting development packages or application components.
% Execution: This metric is best to describe, when quality organization stakeholders want to know how much % of test suite have been covered. Typically, 90-95% test coverage is a very good indicator for any test phase.
Functional coverage is the first & foremost requirement to test a product i.e. product should conform to the requirements it is designed for, with specifying the limitations, if there are any. Additionally, testing using different variants – testing boundary conditions, decision tables, branches etc.; ensures that product is doing well in all conditions, for desired functionalities. On the other hand, while how many code lines (statements) are getting exercised during the test, indicates Code coverage, this alone is not a healthy quality metrics, because a lot of code statement (e.g. Function Module) can be covered by just executing it with some random or unrelated junk values. Here though code lines will still be traversed and the code coverage metric will appear good, the random data chosen for testing will not reflect conformance to the functionality requirements.
Looking at any of above metric alone can give us an incomplete picture of quality of a product. Hence it is important to take both Functional & Code coverage in consideration, while defining the quality metrics or evaluating a product’s quality. These two metrics are further complemented by % Execution which ensures that a minimum number of test (~ 95%) have been executed.
On that note, let me also cover briefly about an important SAP tool which helps in analyzing the test coverage - Coverage Analyzer SCOV. It’s a very powerful system-wide runtime analysis tool for monitoring and analyzing the code covered by test suites such as Automated tests, Manual Tests, Unit tests, integration tests.
Image: Code Snippet from SCOV tool: Green Highlights the code covered with our automated test cases and RED is the additional avenue for coverage.By making use of SCOV coverage results for (Automated /Manual test suite), we can enhance our functional tests in such a way that we cover the untouched statements/paths/messages in code to enhance the quality of our tests and cover more. Using this tool, we are not only targeting enhanced Functional coverage but complementing it with enhanced code coverage too.
Here is an example to enhance the Code coverage by introducing different test conditions:
Initial Code snippet:
First Test condition: Don’t pass the parameters /pass wrong parameters to retrieve the results to traverse the statement NO_RECORD_FOUND.
Test Condition 2: Don’t pass the mandatory parameters completely to traverse PARAMETER_INCOMPELETE condition.
I have spent my 12 years in Quality management, testing SAP products (Industry solutions) in various releases and have experienced that considering these three aspects(metrics) of coverage are very important in defining & evaluating quality of product. It really helps achieving overall higher quality, higher customer satisfaction, higher reliability and lesser rework.
We at Suite Test Engineering, are leveraging this new test approach to ensure "High Quality" product delivery to our end-users and customers.
- SAP Managed Tags:
