Description
A team ships a generated pricing change. The tests pass, coverage is high, the dashboard is green. Three weeks later a customer reports prices out by a factor of ten. The suite has a test for the pricing function with full branch coverage: it calls the function, captures what comes back, and asserts that the result is whatever the function returned. The test was written from the implementation. It asked the code what the price was, and then agreed with it. The bug was there on day one and the test confirmed it on day one.
A test does two separate things. It runs the code, and it states what the answer should have been. Only the first is execution. The second is the definition of correct, and it has to come from somewhere other than the code, because the code is the thing being checked. That is the half a model structurally cannot supply: it can read what your system does and has no access to what it was supposed to do.
Coverage measures the first half and has been quietly standing in for the second for years. The evidence against that substitution is old and was never absorbed. Studies generating many thousands of suites over large systems found that how much code a suite executes is a weak predictor of whether it catches faults, and the stronger flavours of coverage added nothing. It survived because coverage is easy to compute, easy to display, and easy to write into a policy. Then generation made the number trivially cheap to reach.
What does predict protection is asking a different question: if this behaviour changed, would anything fail? Mutation testing asks it directly by introducing faults and checking whether the suite notices. Property-based tests ask it from the specification side, by stating what must always be true. Both force somebody to decide what correct means before the test exists, which is exactly the judgement no tool provides and no coverage target demands.
The five levels
One question decides which rung you are on: does the system itself act on this, or does a human have to choose to? Each rung carries a tell: the thing that would settle it, so the score is something you checked rather than something you agreed with.
- 1
Blind
Nothing describes what the tests are for. There is no coverage figure and no record of which behaviours are protected, so the value of the suite is a matter of opinion. It passes, and nobody can say what that means beyond the code having run.
The tellAsk what would break if somebody changed a boundary condition in your most important function. If the answer is a guess, and nobody offers to go and try it, you are here.
- 2
Tacit
Test quality depends entirely on who wrote the test. Some engineers write assertions from what the system is supposed to do, and their suites catch real regressions. Others, and every agent, write assertions from what the code currently returns. Both kinds sit in the same suite, indistinguishable from outside.
The tellRead the assertions in two recently added test files by different authors. If one specifies behaviour and the other restates the implementation, and the pipeline treats them identically, you are here.
- 3
Visible but inert
Coverage is computed, displayed and often gated on a threshold. What the number cannot say is whether a test would fail if the behaviour changed, so a suite asserting whatever the code returns clears the gate exactly as well as one defining what the code must do.
The tellChange a comparison operator inside a covered function and run the suite. If it stays green while the report still calls that line covered, the gate measures execution rather than protection.
- 4
Acted on
Something mechanical checks whether the tests catch faults. Mutation testing, or property-based tests over stated invariants, runs in the pipeline and fails when the suite would not have noticed a change in behaviour. A test is measured by what it catches rather than by what it touched.
The tellFind the last change your mutation or property gate refused. It failed on its own because the suite missed an introduced fault, rather than a reviewer deciding the tests looked thin that day.
- 5
Durable
The practice holds while the system grows. New modules arrive inside the same fault-detection thresholds, generated tests are held to them like any others, and somebody still states what must always be true before the tests exist, which is the part no tool produces.
The tellLook at where the fault-detection score has moved over a year, and at what happened when a generated suite arrived. A score that stayed the same or improved is durable. Quietly excluded is the practice ending.
What this dimension rests on
The five levels are observed, from client engagements, and no paper produced them. What the research below establishes is why this dimension is worth scoring at all. No links: the citations carry enough to find them, and they stay findable after a link would have rotted.
Inozemtseva and Holmes, Coverage Is Not Strongly Correlated with Test Suite Effectiveness (ICSE 2014)
Generated many thousands of test suites over large systems and found that how much code a suite executes predicts poorly whether it catches faults. Stronger kinds of coverage added no useful prediction either.
Konstantinou, Degiovanni and Papadakis, Do LLMs Generate Test Oracles That Capture the Actual or the Expected Program Behaviour? (2024)
Compared generated assertions against correct and buggy versions of the same code, and found they capture what the program currently does rather than what it should do. Where the code was wrong, the tests agreed with it.
Where this came from
This dimension has a published article behind it, written from client engagements rather than from a greenfield repository.
Read the article on Test meaningfulness →
How to climb this dimension is not written yet. It is the next thing this page grows, and saying so is more useful than a paragraph of generic advice.