Description
Documentation always had a repair mechanism, and it was not in the document. A reader who hit a step that no longer worked asked somebody, got the real answer, and carried on. Some of them even fixed the page. That informal loop is what held the whole thing together, and it quietly disguised how much of it had stopped being true.
An agent cannot ask. It reads your instructions as the state of the world, generates against them at pipeline speed, and produces work that is careful, coherent and aimed at a description that stopped matching reality months ago. The diligence is real. It is pointed at the wrong thing, and nothing in the process is positioned to notice, because the review gate that would have caught it is already carrying more than it used to.
Worse, showing the model the current documentation does not reliably fix it. Studies of what happens when a model's training and the present state of an interface disagree find that stale internal knowledge often wins even when the correct context is supplied. And the divergence between prose and system is not cosmetic in the first place: changes where the description and the code drift apart are more likely to introduce a defect soon after.
The volumes make it structural rather than a matter of diligence. Code now changes faster while documentation is still updated at human pace, so the gap widens mechanically, and generated documentation makes the perceived quality rise at the same time. The only thing that closes it is executing the load-bearing parts rather than reading them: setup and run instructions run from a clean checkout, so drift turns the build red instead of turning up in somebody's second week.
The five levels
One question decides which rung you are on: does the system itself act on this, or does a human have to choose to? Each rung carries a tell: the thing that would settle it, so the score is something you checked rather than something you agreed with.
- 1
Blind
The documentation exists and nothing tests it. No one has followed the setup instructions from a clean machine in months, so whether they still work is unknown, and the people who could tell you already have the environment working and never read them.
The tellTake the README of a service you have never set up yourself and follow its first five steps on a clean machine. If you cannot finish them without asking somebody, and nothing in the pipeline would have caught that, you are here.
- 2
Tacit
Onboarding works because somebody sits with the new joiner and fills in the gaps. The steps that were wrong get fixed for that person and sometimes get written back. The real instructions live in that conversation, which is exactly the channel an agent does not have.
The tellAsk your most recent joiner how many times they had to ask a person to get the system running. If the answer is more than once, the documentation is being repaired in conversation.
- 3
Visible but inert
The documentation is taken seriously. It is reviewed, it is updated when someone notices a gap, and it may look excellent. Nothing runs it, so it drifts silently from the moment of the last edit, and both a new joiner and an agent believe it until something fails further downstream.
The tellTake your setup instructions and diff them against your build configuration. If they disagree anywhere, and both were considered current, nothing is executing the prose.
- 4
Acted on
The load-bearing documentation is executed rather than read. Setup and run steps are run in the pipeline from a clean checkout, and a step that no longer works turns the build red. The prose cannot quietly stop being true, because something tries it on every change.
The tellFind the last time your documentation broke the build. Something ran the instructions and failed on them, rather than a new joiner discovering the gap and mentioning it in their second week.
- 5
Durable
The practice grows with the system. New services arrive with their instructions executed like everything else, documentation generated by a model is verified against the running system before it is trusted, and the checks are not the first thing switched off when the pipeline gets slow.
The tellCount the repositories where the instructions are executed in the pipeline against the total. If new repositories join that set by default, rather than by somebody volunteering, the practice is durable.
What this dimension rests on
The five levels are observed, from client engagements, and no paper produced them. What the research below establishes is why this dimension is worth scoring at all. No links: the citations carry enough to find them, and they stay findable after a link would have rotted.
Ashik, Wang, Chen, Asaduzzaman and Tian, When LLMs Lag Behind: Knowledge Conflicts from Evolving APIs in Code Generation (2026)
Tested what models do when their training and the current documentation disagree, and found stale internal knowledge often wins. Showing a model the correct current context does not reliably override what it already believes.
Radmanesh, Imani, Ahmed and Moshirpour, Investigating the Impact of Code Comment Inconsistency on Bug Introducing (2024)
Studied changes where the prose and the code diverged across many projects, and found those changes more likely to introduce a bug soon afterwards. Stale description is not cosmetic, it misleads the next reader.
Where this came from
This dimension has a published article behind it, written from client engagements rather than from a greenfield repository.
Read the article on Documentation integrity →
How to climb this dimension is not written yet. It is the next thing this page grows, and saying so is more useful than a paragraph of generic advice.