Description
Two organisations adopt the same tools. Same models, same integrations, same enthusiasm, same budget. One sees defects fall and delivery steady. The other watches incidents per change climb and its change failure rate rise. The technology is identical. What differs is underneath it: whether the pipeline can reliably tell code that works from code that does not.
A pipeline that can do that is not a matter of having one. It is a matter of whether the signal is believed. Red has to mean broken, and it stops meaning broken the moment a meaningful share of red is noise. The adaptation that follows is completely rational and almost impossible to reverse: people rerun instead of investigating, and after a while nobody can remember which failures were worth reading.
The damage runs in both directions, which is the part teams usually miss. Studies of industrial pipelines find developers rerunning jobs that had already passed, because green is not trusted either, and real failures sitting quietly inside builds that reported success. A dashboard in that state produces confidence without producing safety, and both halves of that are expensive.
None of this is new, and none of it was fatal while a person read most of what shipped. What changed is the volume arriving at the gate, and the fact that this gate is now the last mechanical thing between generated code and production. So the question this dimension scores is not whether you have continuous integration. It is whether the flake rate is a number somebody owns and something enforces, or a shared understanding that red is sometimes just red.
The five levels
One question decides which rung you are on: does the system itself act on this, or does a human have to choose to? Each rung carries a tell: the thing that would settle it, so the score is something you checked rather than something you agreed with.
- 1
Blind
There is no flake rate, because nothing measures one. Failures are handled by rerunning until the pipeline goes green, and how often that happens is recorded nowhere. The pipeline is a gate people pass through rather than a signal anybody reads.
The tellOpen last month's pipeline history and count the failed runs that were followed by a rerun with no change to the code. If nothing records that, and nobody can give you the share, nothing is counting.
- 2
Tacit
Some engineers know exactly which suites are unreliable, and roughly how many reruns a merge costs on a bad day. They triage failures quickly because of it, and none of that knowledge exists outside them. A new joiner treats every red as real until somebody tells them not to bother.
The tellAsk who decides whether a red build is worth investigating. If the honest answer is a person rather than a rule, the signal is being interpreted rather than trusted.
- 3
Visible but inert
Flakiness is measured. There is a dashboard, a list of the worst offenders, perhaps a cleanup each quarter. Nothing acts on it: flaky tests stay in the blocking suite, reruns stay unlimited, and the number records the loss of trust instead of stopping it.
The tellFind your flakiest test, then find what has happened to it. If it is still in the blocking suite and the last thing anyone did was rerun it, the measurement connects to nothing.
- 4
Acted on
The pipeline defends its own signal. A test that flakes past a threshold is quarantined or fails the build on its own, reruns are counted and capped, and the flake rate is a number somebody is accountable for driving down. Red means broken because the system makes it mean that.
The tellFind the last test that started flaking. Something quarantined it or failed the build over it automatically, and it happened because the rate crossed a threshold, not because an engineer got annoyed enough to act.
- 5
Durable
Trust in the pipeline survives growth. New repositories arrive with the same thresholds, feedback stays fast enough that people wait for it rather than bypassing it, and the flake budget is defended in the week a release is late, instead of switched off for one evening.
The tellLook at the flake rate and the time to a first signal across the last year, as the suite grew. Flat or falling, with new repositories starting inside the same limits, is durable.
What this dimension rests on
The five levels are observed, from client engagements, and no paper produced them. What the research below establishes is why this dimension is worth scoring at all. No links: the citations carry enough to find them, and they stay findable after a link would have rotted.
Aïdasso, Bordeleau and Tizghadam, On the Illusion of Success: An Empirical Study of Build Reruns and Silent Failures in Industrial CI (2024)
Studied build reruns across industrial projects and found developers rerunning jobs that had already passed, and real failures hiding inside successful builds. The signal is unreliable in both directions, not only in the red one.
Gruber and Fraser, A Survey on How Test Flakiness Affects Developers and What Support They Need To Address It (ICST 2022)
Surveyed developers and found flakiness both widespread and behaviour-changing. People stop investigating and start rerunning, and they stop writing tests for the areas that flake, which is how the erosion becomes permanent.
Where this came from
This dimension has a published article behind it, written from client engagements rather than from a greenfield repository.
Read the article on CI reliability →
How to climb this dimension is not written yet. It is the next thing this page grows, and saying so is more useful than a paragraph of generic advice.