This document will help beginners to understand the basics of code coverage, underlying concepts and calculations. As we all are working on code coverage increments and hence , it is very crucial for us to understand the ‘Code Coverage’ topic in an explicit manner so that we can work better on it. Some of the examples listed below are from the area currently I am working on.
Question: What is Code coverage?
Answer: Code Coverage is simply a measure of the code that is tested. Code Coverage is a measurement of how many lines/blocks/arcs of our code are executed while the automated tests are running. Simply put, code coverage is a way of ensuring that our tests are actually testing the code. When we run your tests, we are presumably checking that we are getting the expected results.
Code coverage basically tests that how much of your code is covered under tests. So, if we have 90% code coverage than it means there is 10% of code that is not covered under tests.
A good example will be this:
if (employee. IsHired())
Now, in the above code there are two paths/branches. If we are always hitting the “YES” branch then we are not covering the else part and it will be shown in the Code Coverage results. This will give us idea that what is not covered and we can create an automate to cover the else part.
There are several criteria which can be used to determine how well our tests exercise our code. The simplest one is statement coverage, which simply tells us whether we exercised the statements in our code.
When working with code coverage, and when testing in general, it is wise to remember the following quote from Dijkstra, who said:
Testing never proves the absence of faults, it only shows their presence.
Using code coverage is a way to try to cover more of the testing problem space so that we come closer to proving the absence of faults, or at least the absence of a certain class of faults.
Code coverage is just one weapon in the software engineer’s testing arsenal.
Question: What are Code coverage metrics?
Answer: A number of different metrics are used to determine how well exercised the code is. Some of the most common metrics are listed below:
– Statement coverage : Statement coverage is the most basic form of code coverage. It is used to calculate and measure number of statements in the source code which have been executed. A statement is covered if it is executed.
– Branch coverage :The goal of branch coverage is to ensure that whenever a program can jump, it jumps to all possible destinations. The simple example is shown below:
if (Employee A= Success)
Full coverage is only achieved here only if ‘Employee A’ is true on one occasion and false on another. Branch coverage is also called decision, arc or all edges coverage.
– Path coverage:The purpose of path coverage is to ensure that all paths through the program are taken. In any program, there will be an enormous number of paths through the program. One way to better understand path coverage is to create a graphical diagram of the code flow and follow each of the possible paths through every decision point.
– Condition coverage: When a boolean expression is evaluated it can be useful to ensure that all the terms in the expression are exercised. For example:
if ((A II B) && C)
<< Few Statements>>
In order to ensure complete Condition coverage criteria for the above example, A, B and C should be evaluated at least once against “true” and “false”.
So, in our example, the 3 following tests would be sufficient for 100% Condition coverage testing.
A = true | B = not eval | C = false
A = false | B = true | C = true
A = false | B = false | C = not eval
Condition coverage is also known as expression, condition-decision and multiple decision coverage.
Question: How to depict/analyze code coverage?
Answer: So, we are aware now that code coverage help us to ensure that our software is well tested.
Let’s assume that we have successfully run our test suite with SAP code coverage Analyzer tool, SCOV. We analyze the output from SCOV and it tells us that say we have achieved 80% statement coverage. Now, we look at a more detailed report and notice that a couple of packages have never been called.
We can now cover the untouched packages through innovative ideas/solutions for e.g. we write some automates which test these untouched packages leading to increase in code coverage and thus strengthen our test suite.
Question: What are the benefits of Code Coverage?
Answer: When we extend our automates, we might notice that a certain missing else clause is never tested, so we decide to create an automate to take that branch, but when we execute that automate; the program performs incorrectly. It may be an application error in the implementation. Maybe no one had even considered what should happen for the scenario which we just created to cover that else path.
It helps us to test untouched code lying in the system and thus help us to unearth more defects.
Question: What is the process to carry out code coverage exercise?
1: We should have an explicit user in the backend system say ABC (in which code coverage exercise needs to be performed) which will be used to execute our automates.
2: We should have System Data Container and target system connected to system ABC.
3: Pre-check the system for required Business Functions , services mappings and any other basic settings/customizings.
4: Do not perform any manual tests using the code coverage user.
Question: How to calculate code coverage?
Answer: Let us assume any SAP system where we need to calculate code coverage reading.
1: Launch the SAP code coverage Analyzer tool SCOV using SCOV as transaction.
2: Click on Details button and enter test group( e.g STEC) and package for example say BC*
3: Go to settings tab and Select radio button ->Package and then enter ‘999’ as maximum no. of objects.
4: Click on Execute button
5: Results are displayed in below format:
6: Click on Export icon and select Spreadsheet option, in the pop-up choose option as ‘excel(in office 2007 XLXS Format) and click on continue.
7: It gets downloaded in excel format as below:
8: Now, here we get ‘Current Statement Coverage’ and ‘Total No. of Statements’ in the system:
9: To calculate ‘Total No. of statements covered’, use the formula and capture it in new column say ‘I’ in the downloaded excel:
Total No. of Statements Covered= [Current Statement Coverage(Percent) * No. of Statements] / 100
= [Column D* Column H] / 100
10: To calculate baseline measurement for a respective module, use below formula:
Code Coverage %= [Total No. of Statements covered in a module / Total No. of Statements in a module] *100
11: To calculate baseline measurement for team (all modules), use below formula:
Code Coverage %= [Total No. of Statements covered in all modules/ Total No. of Statements in all modules] * 100
Hence, keeping track of code coverage helps us to answer whether we test enough. It does this by showing the ratio between tested and untested code. It’s interesting to know what code is covered, but it’s far more interesting to know what code is not covered by our tests.