How To: Integrate LCov with PHPUnit to Test PHP Extensions

When writing or debugging PHP extensions, it is very useful to have test coverage. It is also very interesting to know what test covers which files, similar to what PHPUnit generates for PHP:

The lack of test coverage was an issue for me, for example, with Zephir: even though the compiler (PHP-based) has test coverage enabled, the C based kernel does not, and when dealing with bugs you sometimes realize that the test code that is assumed to test a specific issue in the kernel, does not actually test it, as the required branch is not taken.

First of all, you need to build your PHP extension with code coverage enabled:

CPPFLAGS="-DCOVERAGE"
CFLAGS="-O0 --coverage"
LDFLAGS="--coverage"
EXTRA_LDFLAGS='-precious-files-regex \.gcno\$$'
export CPPFLAGS
export CFLAGS
export LDFLAGS
export EXTRA_LDFLAGS
phpize && ./configure && make -B -j$(getconf _NPROCESSORS_ONLN) && make install

-O0 in CFLAGS turns off any optimizations (optimizations are not much compatible with code coverage, as an optimizer is free to reorder statements), --coverage in CFLAGS instructs the compiler to add code so that program flow arcs are instrumented and produce a notes file (.gcno) that code coverage utilities can use to show program coverage; --coverage in LDFLAGS instructs the linker to link the code against libcov (or any other necessary library or libraries). EXTRA_LDFLAGS instructs libtool not to kill notes files, as they are necessary for code coverage generation. Finally, -DCOVERAGE in CPPFLAGS defines a preprocessor macro called COVERAGE (I will later explain why it is necessary).

After the source code is built, you usually run tests (make test or vendor/bin/phpunit or whatever), then lcov (something like lcov --no-external --capture --quiet --output-file coverage.info), and finally genhtml (genhtml -q -o report coverage.info).

This will generate a code coverage report but that report will show the combined information about all tests, and there is no way to tell what test covered which lines.

PHPUnit has a notion of a test listener — code that reacts to such events as “test started”, “test ended”, “test failed”, etc. We can create our own test listener which will invoke lcov after each test to save the coverage information.

Something like this (warning: the code below is not production ready, treat it like an example):

class LCovTestListener implements PHPUnit\Framework\TestListener
{
    use PHPUnit\Framework\TestListenerDefaultImplementation;

    private $base;
    private $dir;
    private $tracedir;

    private $tests = [];

    private static function delTree(string $dir)
    {
        $files = array_diff(scandir($dir), ['.', '..']);
        foreach ($files as $file) {
            $item = $dir . '/' . $file;
            is_dir($item) ? self::delTree($item) : unlink($item);
        }

        return rmdir($dir);
    }

    public function __construct()
    {
        $this->dir       = dirname(__DIR__);
        $this->base      = dirname(__DIR__);
        $this->tracedir  = $this->base . '/coverage/';

        if (is_dir($this->tracedir)) {
            self::delTree($this->tracedir);
        }

        mkdir($this->tracedir);
        passthru(sprintf('lcov --zerocounters --directory %s --quiet', escapeshellarg($this->dir)));
    }

    public function __destruct()
    {
        $args = [];
        foreach ($this->tests as $test) {
            $args[] = '-a ' . escapeshellarg($this->tracedir . $test . '.info');
        }

        $args[]  = '-o ' . escapeshellarg($this->tracedir . 'COMBINED.info');
        $command = 'lcov -q ' . join(' ', $args);
        passthru($command);

        $command = sprintf('genhtml -q -s -o %s %s', escapeshellarg($this->tracedir . 'OUTPUT'), escapeshellarg($this->tracedir . 'COMBINED.info'));
        passthru($command);
    }

    public function endTest(PHPUnit\Framework\Test $test, $time)
    {
        $name  = $test->getName(false);
        $class = get_class($test);

        if (substr($name, -5) === '.phpt') {
            $name = 'PHPT_' . basename($name, '.phpt');
        }
        else {
            $name = $class . '_' . $name;
        }

        $name = str_replace('\\', '_', $name);

        $this->tests[] = $name;
        $command = sprintf(
            'lcov --no-external --capture --directory %s --base %s --test-name %s --quiet --output-file %s',
            escapeshellarg($this->dir),
            escapeshellarg($this->base),
            escapeshellarg($name),
            escapeshellarg($this->tracedir . $name . '.info')
        );

        passthru($command);
        // Tests cover different file, make sure to zero all counters in order not to affect subsequent tests
        passthru(sprintf('lcov --zerocounters --directory %s --quiet', escapeshellarg($this->dir)));
    }
}

In brief:

• this code is for PHPUnit 6; for PHPUnit 7 you will need to modify function prototypes slightly;
• __construct() initializes all necessary variables (it assumes that the extension source directory is one level above the location of the file, hence dirname(__DIR__) — adjust accordingly if this is not the case for your setup; what is dir and base, you can read here, but most likely they will be the same and point to the top level source directory of your extension; tracedir is the directory to which tracefiles will be saved), and runs lcov --zerocounters to zero all counters after previous runs;
• endTest() is invoked when a particular test finishes. It makes the name of the test compatible with what lcov expects (the manual says that Valid test names can consist of letters, decimal digits and the underscore character), then runs lcov to generate a tracefile for the given test, and then runs lcov again to zero all counters. It is necessary to zero all counters after each test to get the correct results. Imagine that you have two tests which test func1() and func2() respectively. The first test will correctly generate the coverage for func1(). But if the counters are not reset, the second test will show that it covered func1() as well, which may not the case;
• __destruct combines all tracefiles into the “summary” file (lcov -q -a tracefile1 -a tracefile2 -o COMBINED.info) and then runs genhtml to produce the report.

OK, now we add our test listener to phpunit.xml:

<listeners>
	<listener class="LCovTestListener" file="./unittests/LCovTestListener.php"/>
</listeners>

and ready to get the test coverage.

Unfortunately, no. If you run PHPUnit, you will see something like this:

geninfo: WARNING: no .gcda files found in /path/to/source - skipping!

This happens because .gcda files are flushed when the application terminates. However, there is a way to tell it to flush the data right now. But this requires some changes to the extension code.

Please note that this code should be available only when the extension is built with profiling information. This is where -DCOVERAGE from CPPFLAGS above comes into play.

First we need to define a prototype for __gcov_flush() function:

#ifdef COVERAGE
extern void __gcov_flush(void);
#endif

Then we need to declare a PHP function which will call __gcov_flush:

#ifdef COVERAGE
static PHP_FUNCTION(flush_coverage)
{
    __gcov_flush();
}
#endif

Finally, we need to reference that function from the extension function table:

static const zend_function_entry fe[] = {
#ifdef COVERAGE
    ZEND_FE(flush_coverage, NULL)
#endif
    /* ... */
    ZEND_FE_END
};

zend_module_entry EXTENSION_module_entry = {
    STANDARD_MODULE_HEADER,
    EXTENSION_NAME,
    fe,
/* ... */
};

Rebuild the extension and install it.

Now we need to update our test listener to flush the coverage: add this as the first line in endTest():

flush_coverage();

To make the test listener usable regardless of whether the extension is built with profiling information, you can modify the code like this:

    public function __construct()
    {
        $this->enabled = function_exists('flush_coverage');

        if ($this->enabled) {
            /* ... */
        }
    }

    public function __destruct()
    {
        if ($this->enabled) {
            /* ... */
        }
    }

    public function endTest(PHPUnit\Framework\Test $test, $time)
    {
        if ($this->enabled) {
            flush_coverage();
            /* ... */
        }
    }

Now, if you run the tests, and open index.html from coverage/OUTPUT/, you will see something like this:

Bad news is that you LCov is (yet?) incapable of showing which tests covered a specific line (like PHPUnit on the screenshot above), it only shows the number of times a particular line was hit, like this:

However, we do have coverage information for every test we had run. Say, if I want to see the coverage for PHPT_container_pimple_002, I can run something like this:

genhtml -q -o PHPT_container_pimple_002 PHPT_container_pimple_002.info

and then I will have the requested coverage available.

I agree that this is not the best solution but this is what is available so far.

Tracefile format is pretty simple (and for PHP there is a reader available), and it contains enough information to produce the report. Therefore in theory it should not be difficult to write a replacement for genhtml but this is work in progress.