Each time we talk about testing, we also talk about code coverage. But why is it important to keep an eye on code coverage and what are possible code coverage goals?
What is code coverage?
Code coverage is the sum of all coverage goals that are covered by a set of test cases. Coverage goals are e.g. statements or decisions. The ISO 26262 names coverage goals like statement, decision/branch and modified condition/decision coverage for unit testing. Depending on the risk class (=ASIL level) a subset of these coverage goals are highly recommended. Nevertheless, these coverage goals are just a subset of the available coverage goals for C Code in general. There are more coverage goals like function, function call, switch case, relation operator coverage. In addition, it is useful to consider functional scenarios, derive equivalence classes, boundary value checks or individually specified critical situations. For all the coverage goals the tester wants to achieve a high (if possible) 100% coverage. Beside the code coverage goals named before, we should consider robustness goals like division-by-zero, critical down casts, range violations and unreachability as well.
It is not easy to reach high coverage results with handwritten test cases. However, based on a technology called Model Checking it is possible to generate tests automatically and to create a mathematical proof, that a certain coverage or robustness goal can never be reached. As a result, all available coverage goals are analyzed and get reported as either covered or unreachable.
Let’s have a look on some samples of typical coverage goals. Coverage goals can have a different amount of properties that can either be covered or unreachable. For all of the coverage goals high coverage should be achieved.
Now let’s have a look at robustness goals. For these coverage goals an unreachable result is expected. Otherwise this might lead to unintended behavior.
Of course, not all projects focus on all of these coverage and robustness goals but a sufficient subset that fulfills the project safety goals. In general, there are three levels test goal are related to:
- Unit Testing level with a detailed focus on the implementation and complex test goals like MC/DC.
- Integration level that focuses on the interaction of several units and components
- Architecture level that focuses on the whole software project and
What test goals are taken into account for each level is based on the risk class of the component and is usually defined by the customer and the quality assurance.
