In an IRC discussion, the topic of collecting and using reverse test coverage information came up.
First some definitions:
Forward test coverage answers the question:
Given these tests, what part of the codebase gets executed?
Reverse test coverage answers the question:
Given this part of the codebase, which tests execute it?
Most conventional test coverage tools answer the "forward" coverage question: developers want to know what code the tests exercise or miss.
Reverse coverage, by contrast, is relatively specialised. One main application of it is in the context of big, complex, automated integration builds, where you only want to re-run the subset of tests that depend on the code that changed with each commit, rather than the full test suite.
As a result, reverse coverage support tends to be obscure, or locked away inside heavy-weight distributed build systems (such as Google's formerly in-house Bazel).
However, reverse coverage is far from being useful only at that big scale. Imagine being able to:
Those features could be baked into existing tools and test runners (unittest, py.test, trial, …), but that might involve a lot of redundant work.
Python already has a de facto standard generic code coverage tool, coverage.py. One of the great things about it that it can invoke any Python tool or script and collect configurable coverage information about its execution, without any special support from the tool being invoked.
Could this same generic approach work for reverse coverage?
Update: coverage.py now has support for capturing this!
https://python-tia.readthedocs.io/ – The generic Test Impact Analysis (TIA) preprocessor for test tools.
This project provides this useful overview of related tools:
https://github.com/tarpas/pytest-testmon
This pytest plugin looks like a solid and user-friendly implementation of the idea for faster TDD and CI.
Another list of related work:
https://github.com/tarpas/pytest-testmon/issues/42#issuecomment-310487641