100% Code Coverage

I have just seen a funny picture:

Well, it’s not very difficult to achieve 100% line coverage (unless you are writing highly fault-tolerant code, and not all error conditions are possible to reproduce in the test environment). It is more difficult to produce 100% branch coverage. And it is much more difficult (if at all possible) to ensure that all possible code paths are covered.

While 100% looks nice, if a developer cares only about covering all lines, 100% coverage gives false sense of security.

For example, below is a (greatly) simplified code from a real project (the real code was much more complex and had more conditions, therefore the evident bug was not that evident):

$sql   = 'SELECT * FROM table WHERE '
$where = '';

if (condition1()) {
    $where .= 'something = "something"';
}

if (condition2()) {
    $where .= 'something = "something else"';
}

$sql .= $where;
return $db->query($sql);

We have the following set of tests:

• Test 1: condition1() is true, condition2() is true
• Test 2: condition1() is true, condition2() is false
• Test 3: condition1() is false, condition2() is true

It is easy to see that this set of test a) covers all lines, b) covers all branches. Therefore the coverage for this method is 100%.

However, these tests do not cover all possible code paths: when both condition1() and condition2() are false, the resulting SQL query will be invalid. Thus, we need either one more test or to rewrite a code so that the query always remains valid.

Thus, if a tested method has only two conditions, we need 2²=4 tests; if it has three conditions, we need 2³=8 tests; for 10 conditions, the number of tests will be 1024. And if conditions are complex, the number of tests to cover all possible code paths will increase (in practice you usually do not need so many tests: for example, according to the NIST Structured Testing methodology, the cyclomatic complexity of a module determines the number of white-box tests that are required to obtain sufficient coverage of the module).

So, is the code coverage important? Sure it is: to rephrase Dick Brandon, it is like sex; when it’s good, it’s very, very good, and when it’s bad, it’s better than nothing. But code coverage alone is not enough to estimate maintainability of the code. Personally I would prefer to spend more time on decreasing the cyclomatic complexity of the code than writing tests to cover all trivial getters and setters.