The AI Readiness self-assessment
You left one unfinished
You have scored 0 of 20 on this device, and nothing expired while you were away. Answers here never do.
Starting fresh keeps the old attempt on this device rather than deleting it.
Before question one
What are you scoring?
A score with no scope is a number without units, so this is the one thing to settle before question one. It is not an administrative step: it decides what the number means.
Widening the scope should lower your score, because the worst case gets worse. Across a business unit with a modernisation half done, that usually means a floor of 1 on most of the twenty. True, and close to useless.
If you are halfway through a modernisation, do not do one assessment. Do two. Score the platform you have built. Then score the estate it has not replaced yet. The distance between those two numbers is where your readiness boundary actually lies, and it is worth more than either number on its own.
It appears on your report so you know which system you scored. It stays on this device and is never transmitted, because it is the one thing you could type here that would identify your organisation.
A little about you (all optional)
Nothing here changes your result or your questions. It is what lets a future version of this say which dimensions floor for a bank of five thousand people rather than for software companies in general. Every band is deliberately wide, none of it is transmitted unless you finish, and there is no field here that could name your organisation. Skip any of it.
Two rules before you start. If you do not know the answer to a question, that is a 1. Not knowing is the level, not a gap in the instrument.
And if you are halfway through a modernisation, you will recognise your organisation in two levels at once: the new platform on one, the estate it has not replaced yet on another. Both are true. Score the worse one, because that is the one an agent reaches first.
Foundation & Structure
Question 1 of 20
Repository structure
Your repository setup is the map an agent works from: how many repositories there are, where the boundaries between them run, and whether anything except somebody's memory keeps work on the right side of them. It has no picture of your estate other than the one it can read.
At its best the machine holds the shape, whichever model you run: scoped builds and enforced module boundaries in a monorepo, versioned and enforced contracts across many repositories, so crossing a boundary fails rather than merely being noticed. At its worst the structure accumulated rather than being chosen, is written down and nothing more, and holds only for as long as the right person reads the pull request.
Across everything in your scope, does your repository setup help an agent reason about the system, or force it to guess?
The research
- Luo et al., RepoCod: Can Language Models Replace Programmers? (ACL 2025)
- Zhang et al., LLM Hallucinations in Practical Code Generation (ACM PACMSE 2025)
Answer for the worst case in your scope, not the typical one. Each level carries a tell: the thing that would settle it, so the score is something you checked rather than something you agreed with. At levels 4 and 5 that is usually a command you can run this afternoon. At levels 1 and 2 it is usually a question for two colleagues, because the absence of a measurement is not something anyone can go and look at.
If nobody in your scope can produce it, that is a 1. Not knowing is the level, not a gap in the instrument. If the dimension itself is unfamiliar, read it at length first.
The wording of these levels changed since you answered this one. Your answer is still here. Confirm it or change it.
Choose the level you recognise. There is no skip here: the verdict is the lowest of your scores, and a dimension you never scored cannot be part of it. If you do not know, that is a 1. If the question is unanswerable because your scope is too wide, rather than because your system has no answer, a narrower assessment will tell you more than this one.
Foundation & Structure
Question 2 of 20
Cognitive load per module
This is how much of your system somebody has to read in order to change one thing. A module that needs five others read first costs a new engineer a week, and costs an agent its whole context window every session, because unlike your engineers it never builds the familiarity that makes a tangled module survivable.
At its best coupling and complexity carry a budget the build enforces, and that budget survives the release that wants to breach it. At its worst there is no number, and the only signal is which parts of the system people quietly avoid.
Does anything in your scope know how tangled its worst module is, or only your people?
The research
- He, Miller, Agarwal, Kaestner and Vasilescu, Speed at the Cost of Quality: How Cursor AI Increases Complexity (MSR 2026)
- Rando et al., LongCodeBench: Evaluating Coding LLMs at 1M Context Windows (2025)
Answer for the worst case in your scope, not the typical one. Each level carries a tell: the thing that would settle it, so the score is something you checked rather than something you agreed with. At levels 4 and 5 that is usually a command you can run this afternoon. At levels 1 and 2 it is usually a question for two colleagues, because the absence of a measurement is not something anyone can go and look at.
If nobody in your scope can produce it, that is a 1. Not knowing is the level, not a gap in the instrument. If the dimension itself is unfamiliar, read it at length first.
The wording of these levels changed since you answered this one. Your answer is still here. Confirm it or change it.
Choose the level you recognise. There is no skip here: the verdict is the lowest of your scores, and a dimension you never scored cannot be part of it. If you do not know, that is a 1. If the question is unanswerable because your scope is too wide, rather than because your system has no answer, a narrower assessment will tell you more than this one.
Foundation & Structure
Question 3 of 20
Domain language consistency
This is whether one concept has one name. An agent learns your vocabulary from your code and from nowhere else, so when four services call the same thing four different things, it learns all four as equally correct and then produces a fifth.
At its best the glossary is machine-checked: a banned synonym fails a lint rule, and a new public name is checked against the vocabulary before it merges. At its worst there is an accurate glossary that took real work, sitting on a wiki page nothing in the pipeline has ever opened.
If you asked three teams anywhere in your scope to define your most contested domain word, would you get the same answer?
The research
- Fakhoury et al., Cognitive Load from Linguistic Antipatterns, measured with fNIRS brain imaging (ICPC 2018, Distinguished Paper)
- Aghajani et al., Linguistic Antipatterns in APIs (2018)
Answer for the worst case in your scope, not the typical one. Each level carries a tell: the thing that would settle it, so the score is something you checked rather than something you agreed with. At levels 4 and 5 that is usually a command you can run this afternoon. At levels 1 and 2 it is usually a question for two colleagues, because the absence of a measurement is not something anyone can go and look at.
If nobody in your scope can produce it, that is a 1. Not knowing is the level, not a gap in the instrument. If the dimension itself is unfamiliar, read it at length first.
The wording of these levels changed since you answered this one. Your answer is still here. Confirm it or change it.
Choose the level you recognise. There is no skip here: the verdict is the lowest of your scores, and a dimension you never scored cannot be part of it. If you do not know, that is a 1. If the question is unanswerable because your scope is too wide, rather than because your system has no answer, a narrower assessment will tell you more than this one.
Foundation & Structure
Question 4 of 20
Semantic drift
Names fragment under change, in every system, always. Refactoring is the work that used to slow that down, and it is the first thing dropped when a deadline approaches. So this dimension is about the rate rather than the state: not how consistent the names are today, but which way they have been moving since last year.
At its best drift is tracked over time and the trend is what gets consolidation scheduled as delivery rather than requested as a favour. At its worst nobody could say whether the system is more coherent than it was a year ago or less, so the question gets settled by whoever sounds most certain.
Does the same concept have one name across your scope or several, and do you know whether it is worse than a year ago?
The research
- Lehman, Laws of Software Evolution (IEEE 1980)
- Wang et al., Deep Learning-Based Identification of Inconsistent Method Names (Empirical Software Engineering, 2025)
Answer for the worst case in your scope, not the typical one. Each level carries a tell: the thing that would settle it, so the score is something you checked rather than something you agreed with. At levels 4 and 5 that is usually a command you can run this afternoon. At levels 1 and 2 it is usually a question for two colleagues, because the absence of a measurement is not something anyone can go and look at.
If nobody in your scope can produce it, that is a 1. Not knowing is the level, not a gap in the instrument. If the dimension itself is unfamiliar, read it at length first.
The wording of these levels changed since you answered this one. Your answer is still here. Confirm it or change it.
Choose the level you recognise. There is no skip here: the verdict is the lowest of your scores, and a dimension you never scored cannot be part of it. If you do not know, that is a 1. If the question is unanswerable because your scope is too wide, rather than because your system has no answer, a narrower assessment will tell you more than this one.
Truth Signals
Question 5 of 20
CI reliability
A pipeline is a safety net only while people believe it. When red sometimes means broken and sometimes means unlucky, teams learn to press rerun, and they learn it long before AI arrives. Then the volume of changes goes up and that habit is the last thing standing between generated code and production.
At its best the flake rate is measured, driven down and enforced, so a red build is treated as information. At its worst nobody knows the flake rate, reruns are uncounted, and green gets rerun too, because the signal is not trusted in either direction.
Somewhere in your scope a pipeline went red this week. Did anybody investigate it, or did they press rerun, and would you defend that choice?
The research
- Aïdasso, Bordeleau and Tizghadam, On the Illusion of Success: An Empirical Study of Build Reruns and Silent Failures in Industrial CI (2024)
- Gruber and Fraser, A Survey on How Test Flakiness Affects Developers and What Support They Need To Address It (ICST 2022)
Answer for the worst case in your scope, not the typical one. Each level carries a tell: the thing that would settle it, so the score is something you checked rather than something you agreed with. At levels 4 and 5 that is usually a command you can run this afternoon. At levels 1 and 2 it is usually a question for two colleagues, because the absence of a measurement is not something anyone can go and look at.
If nobody in your scope can produce it, that is a 1. Not knowing is the level, not a gap in the instrument. If the dimension itself is unfamiliar, read it at length first.
The wording of these levels changed since you answered this one. Your answer is still here. Confirm it or change it.
Choose the level you recognise. There is no skip here: the verdict is the lowest of your scores, and a dimension you never scored cannot be part of it. If you do not know, that is a 1. If the question is unanswerable because your scope is too wide, rather than because your system has no answer, a narrower assessment will tell you more than this one.
Truth Signals
Question 6 of 20
Test meaningfulness
A test does two things: it runs the code, and it says what the answer should have been. Coverage measures the first. Protection comes from the second, and that is the half a model cannot supply, because it can read what your code does and not what it was meant to do.
At its best something mechanical checks whether the tests catch faults, and the suite is judged on that rather than on how much code it touched. At its worst coverage is high, generated tests assert whatever the implementation happens to return, and the dashboard stays green over a regression.
Take a function anywhere in your scope, move a boundary by one or flip a comparison, and run the suite. Would anything fail, or would it stay green?
The research
- Inozemtseva and Holmes, Coverage Is Not Strongly Correlated with Test Suite Effectiveness (ICSE 2014)
- Konstantinou, Degiovanni and Papadakis, Do LLMs Generate Test Oracles That Capture the Actual or the Expected Program Behaviour? (2024)
Answer for the worst case in your scope, not the typical one. Each level carries a tell: the thing that would settle it, so the score is something you checked rather than something you agreed with. At levels 4 and 5 that is usually a command you can run this afternoon. At levels 1 and 2 it is usually a question for two colleagues, because the absence of a measurement is not something anyone can go and look at.
If nobody in your scope can produce it, that is a 1. Not knowing is the level, not a gap in the instrument. If the dimension itself is unfamiliar, read it at length first.
The wording of these levels changed since you answered this one. Your answer is still here. Confirm it or change it.
Choose the level you recognise. There is no skip here: the verdict is the lowest of your scores, and a dimension you never scored cannot be part of it. If you do not know, that is a 1. If the question is unanswerable because your scope is too wide, rather than because your system has no answer, a narrower assessment will tell you more than this one.
Truth Signals
Question 7 of 20
Change absorption
Absorption is what happens after the merge: whether the change stays, whether a reviewer understood it, whether anyone outside engineering found out it happened. AI raises the amount of change arriving and leaves the capacity to absorb it exactly where it was, so the share of work that has to be done again is the first thing to rise.
At its best churn and revert rates are tracked on a cadence, and the number is allowed to stop a plan. At its worst code goes in, some of it quietly comes back out, and the effort spent on the second part is invisible to everybody deciding how much more to generate.
Of everything merged anywhere in your scope last month, how much was reverted or rewritten within two weeks, and is that share rising?
The research
- Nagappan and Ball, Use of Relative Code Churn Measures to Predict System Defect Density (ICSE 2005)
- Cohen and Levinthal, Absorptive Capacity: A New Perspective on Learning and Innovation (Administrative Science Quarterly, 1990)
Answer for the worst case in your scope, not the typical one. Each level carries a tell: the thing that would settle it, so the score is something you checked rather than something you agreed with. At levels 4 and 5 that is usually a command you can run this afternoon. At levels 1 and 2 it is usually a question for two colleagues, because the absence of a measurement is not something anyone can go and look at.
If nobody in your scope can produce it, that is a 1. Not knowing is the level, not a gap in the instrument. If the dimension itself is unfamiliar, read it at length first.
The wording of these levels changed since you answered this one. Your answer is still here. Confirm it or change it.
Choose the level you recognise. There is no skip here: the verdict is the lowest of your scores, and a dimension you never scored cannot be part of it. If you do not know, that is a 1. If the question is unanswerable because your scope is too wide, rather than because your system has no answer, a narrower assessment will tell you more than this one.
Truth Signals
Question 8 of 20
Technical debt
Technical debt is the work you already owe on the system you already have. AI does not pay it down. It draws on that debt faster, because generated code lands in the same tangles and the refactor that would have cleaned them up is the first thing dropped under delivery pressure.
At its best the debt is quantified in money, and that figure sits inside the plan for the acceleration rather than beside it. At its worst it is an engineering complaint, real to everyone who touches the code and absent from every document where the money is decided.
Somewhere in your scope is a system everybody agrees needs fixing. Does your AI plan carry a line item for fixing it, or an assumption that the acceleration will pay for it later?
The research
- METR, Measuring the Impact of Early-2025 AI on Experienced Open-Source Developer Productivity (2025)
- GitClear, AI Copilot Code Quality: 2025 Data Suggests 4x Growth in Code Clones (2025)
Answer for the worst case in your scope, not the typical one. Each level carries a tell: the thing that would settle it, so the score is something you checked rather than something you agreed with. At levels 4 and 5 that is usually a command you can run this afternoon. At levels 1 and 2 it is usually a question for two colleagues, because the absence of a measurement is not something anyone can go and look at.
If nobody in your scope can produce it, that is a 1. Not knowing is the level, not a gap in the instrument. If the dimension itself is unfamiliar, read it at length first.
The wording of these levels changed since you answered this one. Your answer is still here. Confirm it or change it.
Choose the level you recognise. There is no skip here: the verdict is the lowest of your scores, and a dimension you never scored cannot be part of it. If you do not know, that is a 1. If the question is unanswerable because your scope is too wide, rather than because your system has no answer, a narrower assessment will tell you more than this one.
Flow & Throughput
Question 9 of 20
Trunk-based development
Every day a branch stays open, the distance between it and the trunk grows, and the cost of closing that distance grows with it. AI raised how much code changes in a day and left most branching strategies exactly where they were, so the merge that used to be fine is now a two-day job with real logic in it.
At its best branches live hours, because feature flags and fast tests make that possible and a rule makes it required. At its worst nobody knows the oldest branch, integration is an event near a deadline, and the conflicts arrive long after the person who wrote the code stopped remembering it.
How old is the oldest branch alive anywhere in your scope right now, and what would it cost to merge it today?
The research
- Kasi and Sarma, An Empirical Investigation into Merge Conflicts and Their Effect on Software Quality (Empirical Software Engineering, 2013)
- Ghiotto, Murta, Barros and van der Hoek, On the Nature of Merge Conflicts: A Study of 2,731 Open Source Java Projects (IEEE Transactions on Software Engineering, 2018)
Answer for the worst case in your scope, not the typical one. Each level carries a tell: the thing that would settle it, so the score is something you checked rather than something you agreed with. At levels 4 and 5 that is usually a command you can run this afternoon. At levels 1 and 2 it is usually a question for two colleagues, because the absence of a measurement is not something anyone can go and look at.
If nobody in your scope can produce it, that is a 1. Not knowing is the level, not a gap in the instrument. If the dimension itself is unfamiliar, read it at length first.
The wording of these levels changed since you answered this one. Your answer is still here. Confirm it or change it.
Choose the level you recognise. There is no skip here: the verdict is the lowest of your scores, and a dimension you never scored cannot be part of it. If you do not know, that is a 1. If the question is unanswerable because your scope is too wide, rather than because your system has no answer, a narrower assessment will tell you more than this one.
Flow & Throughput
Question 10 of 20
Velocity and the constraint
Your delivery pipeline is a chain of stages and one of them has the least capacity. Double what arrives and that stage saturates first. Relieve it and the limit moves to the next. Waiting does not grow in proportion to load either: once a stage is busy, a modest increase produces a queue out of all proportion to it.
At its best the constraint is named, work in progress is capped by tooling that refuses new work past the line, and the cap moves when the constraint does. At its worst every stage looks busy, nobody can say which one is the limit, and capacity gets added where it is easiest rather than where it would help.
If merge-request volume doubled tomorrow anywhere in your scope, which stage fills up first? And do you know that, or are you guessing?
The research
- Reinertsen, The Principles of Product Development Flow (book, 2009)
Answer for the worst case in your scope, not the typical one. Each level carries a tell: the thing that would settle it, so the score is something you checked rather than something you agreed with. At levels 4 and 5 that is usually a command you can run this afternoon. At levels 1 and 2 it is usually a question for two colleagues, because the absence of a measurement is not something anyone can go and look at.
If nobody in your scope can produce it, that is a 1. Not knowing is the level, not a gap in the instrument. If the dimension itself is unfamiliar, read it at length first.
The wording of these levels changed since you answered this one. Your answer is still here. Confirm it or change it.
Choose the level you recognise. There is no skip here: the verdict is the lowest of your scores, and a dimension you never scored cannot be part of it. If you do not know, that is a 1. If the question is unanswerable because your scope is too wide, rather than because your system has no answer, a narrower assessment will tell you more than this one.
Flow & Throughput
Question 11 of 20
Organizational design
Your org chart decides which changes are cheap and which need three handoffs. AI raises what each team produces on its own, and it does nothing at all to the seams between them, so a feature crossing four teams gets faster inside each team and no faster between them.
At its best the boundaries are watched with numbers, they move when the numbers say so, and anyone who understands the domain can review a change inside it. At its worst ownership is fixed, every crossing waits for the one team allowed to touch that component, and the structure is revisited only in a reorganisation.
Pick a recent feature that crossed team boundaries anywhere in your scope. If AI doubled what every one of those teams produced tomorrow, would it arrive sooner, or would the queues between them simply grow?
The research
- MacCormack, Baldwin and Rusnak, Exploring the Duality between Product and Organizational Architectures (Research Policy, 2012)
- Nagappan, Murphy and Basili, The Influence of Organizational Structure on Software Quality (ICSE 2008)
Answer for the worst case in your scope, not the typical one. Each level carries a tell: the thing that would settle it, so the score is something you checked rather than something you agreed with. At levels 4 and 5 that is usually a command you can run this afternoon. At levels 1 and 2 it is usually a question for two colleagues, because the absence of a measurement is not something anyone can go and look at.
If nobody in your scope can produce it, that is a 1. Not knowing is the level, not a gap in the instrument. If the dimension itself is unfamiliar, read it at length first.
The wording of these levels changed since you answered this one. Your answer is still here. Confirm it or change it.
Choose the level you recognise. There is no skip here: the verdict is the lowest of your scores, and a dimension you never scored cannot be part of it. If you do not know, that is a 1. If the question is unanswerable because your scope is too wide, rather than because your system has no answer, a narrower assessment will tell you more than this one.
Flow & Throughput
Question 12 of 20
Code ownership and review routing
Review is where generated code meets human judgement, and routing decides whose. When every change in an area goes to the one person who owns it, tripling the volume does not triple that person. It lengthens a queue, and then it lowers the standard of what comes out of the queue.
At its best routing spreads review across people who genuinely know the area, and queue depth is watched, so a growing wait is caught before review becomes approval without reading. At its worst one name sits in a file, the queue behind it is invisible, and nobody learns how narrow it was until that person is suddenly unavailable.
If the busiest team in your scope vanished tomorrow, how many open changes would have nobody left qualified to review them?
The research
- Bird, Nagappan, Murphy, Gall and Devanbu, Don't Touch My Code! Examining the Effects of Ownership on Software Quality (ESEC/FSE 2011)
- Thongtanunam, McIntosh, Hassan and Iida, Revisiting Code Ownership and Its Relationship with Software Quality in the Scope of Modern Code Review (ICSE 2016)
Answer for the worst case in your scope, not the typical one. Each level carries a tell: the thing that would settle it, so the score is something you checked rather than something you agreed with. At levels 4 and 5 that is usually a command you can run this afternoon. At levels 1 and 2 it is usually a question for two colleagues, because the absence of a measurement is not something anyone can go and look at.
If nobody in your scope can produce it, that is a 1. Not knowing is the level, not a gap in the instrument. If the dimension itself is unfamiliar, read it at length first.
The wording of these levels changed since you answered this one. Your answer is still here. Confirm it or change it.
Choose the level you recognise. There is no skip here: the verdict is the lowest of your scores, and a dimension you never scored cannot be part of it. If you do not know, that is a 1. If the question is unanswerable because your scope is too wide, rather than because your system has no answer, a narrower assessment will tell you more than this one.
Memory & Intent
Question 13 of 20
Decision memory (ADRs)
An architectural decision is two things: the shape of the system, and the reason it has that shape. Only the first survives in the code. An agent asked to add a feature takes the cheapest route, straight across the boundary you drew, and it has no way to know the boundary was expensive, because the reason lived in a meeting.
At its best decisions sit in the repository beside the code, dated, superseded when they change, and a reviewer can point at one. At its worst they exist as slides, a chat thread and one person's memory, and the first change that finds them inconvenient quietly reverses them.
Pick an architectural decision taken anywhere in your scope in the last two years. Can somebody find the reasoning in the repository, or only in the person who made it?
The research
- Robillard et al., Turnover-Induced Knowledge Loss in Practice (ESEC/FSE 2021)
- Storey, From Technical Debt to Cognitive and Intent Debt (2026)
Answer for the worst case in your scope, not the typical one. Each level carries a tell: the thing that would settle it, so the score is something you checked rather than something you agreed with. At levels 4 and 5 that is usually a command you can run this afternoon. At levels 1 and 2 it is usually a question for two colleagues, because the absence of a measurement is not something anyone can go and look at.
If nobody in your scope can produce it, that is a 1. Not knowing is the level, not a gap in the instrument. If the dimension itself is unfamiliar, read it at length first.
The wording of these levels changed since you answered this one. Your answer is still here. Confirm it or change it.
Choose the level you recognise. There is no skip here: the verdict is the lowest of your scores, and a dimension you never scored cannot be part of it. If you do not know, that is a 1. If the question is unanswerable because your scope is too wide, rather than because your system has no answer, a narrower assessment will tell you more than this one.
Memory & Intent
Question 14 of 20
Requirements quality
This is what a written work item leaves an agent to invent. An agent does not ask clarifying questions the way a developer does: it fills the gap with the most likely reading and writes confident, plausible code against it. The vaguer the item, the more confidently wrong the output, and the faster it arrives.
At its best an item cannot be pulled until it is a testable spec, and the check runs before the work starts rather than at review. At its worst items are a title and a sentence of context, and what finished means lives in a conversation that has not happened yet.
Could an agent pick up the top item of any backlog in your scope and build the right thing without guessing?
The research
- Wu et al., ClarifyCoder: Clarification-Aware Fine-Tuning for Programmatic Problem Solving (2025)
- Li, Wu and Chang, ClarEval: A Benchmark for Evaluating Clarification Skills of Code Agents (2026)
Answer for the worst case in your scope, not the typical one. Each level carries a tell: the thing that would settle it, so the score is something you checked rather than something you agreed with. At levels 4 and 5 that is usually a command you can run this afternoon. At levels 1 and 2 it is usually a question for two colleagues, because the absence of a measurement is not something anyone can go and look at.
If nobody in your scope can produce it, that is a 1. Not knowing is the level, not a gap in the instrument. If the dimension itself is unfamiliar, read it at length first.
The wording of these levels changed since you answered this one. Your answer is still here. Confirm it or change it.
Choose the level you recognise. There is no skip here: the verdict is the lowest of your scores, and a dimension you never scored cannot be part of it. If you do not know, that is a 1. If the question is unanswerable because your scope is too wide, rather than because your system has no answer, a narrower assessment will tell you more than this one.
Memory & Intent
Question 15 of 20
Security assumptions
Most security lives in people. The engineer who knows tokens never go in local storage, the architect who remembers why this service signs its own requests. A model has no onboarding and no corridor conversations. It reads your code, and if three patterns for the same thing are in there, one secure and two historical, it treats all three as examples.
At its best the assumption exists once, as a canonical pattern with a scan that fails the build when something deviates. At its worst it is genuinely believed, genuinely important, and written nowhere a machine could find it.
Pick a security assumption your teams hold anywhere in this scope. Where in the codebase would an agent learn it?
The research
- Perry, Srivastava, Kumar and Boneh, Do Users Write More Insecure Code with AI Assistants? (ACM CCS 2023)
- Pearce, Ahmad, Tan, Dolan-Gavitt and Karri, Asleep at the Keyboard? Assessing the Security of GitHub Copilot's Code Contributions (IEEE Symposium on Security and Privacy, 2022)
Answer for the worst case in your scope, not the typical one. Each level carries a tell: the thing that would settle it, so the score is something you checked rather than something you agreed with. At levels 4 and 5 that is usually a command you can run this afternoon. At levels 1 and 2 it is usually a question for two colleagues, because the absence of a measurement is not something anyone can go and look at.
If nobody in your scope can produce it, that is a 1. Not knowing is the level, not a gap in the instrument. If the dimension itself is unfamiliar, read it at length first.
The wording of these levels changed since you answered this one. Your answer is still here. Confirm it or change it.
Choose the level you recognise. There is no skip here: the verdict is the lowest of your scores, and a dimension you never scored cannot be part of it. If you do not know, that is a 1. If the question is unanswerable because your scope is too wide, rather than because your system has no answer, a narrower assessment will tell you more than this one.
Memory & Intent
Question 16 of 20
Documentation integrity
Documentation used to have a repair mechanism built into its reader. A person who hit a step that no longer worked asked somebody, and the asking quietly held the whole thing together. An agent cannot ask. It reads your instructions as ground truth and generates confidently against a description that stopped matching reality in March.
At its best the load-bearing instructions are executed in the pipeline from a clean checkout, so drift turns the build red. At its worst the README is plausible, confident and four months stale, and nothing in the organisation would notice.
If a new developer or an agent followed the README of any repository in your scope from a clean machine, would they get a running system, and would you bet on it?
The research
- Ashik, Wang, Chen, Asaduzzaman and Tian, When LLMs Lag Behind: Knowledge Conflicts from Evolving APIs in Code Generation (2026)
- Radmanesh, Imani, Ahmed and Moshirpour, Investigating the Impact of Code Comment Inconsistency on Bug Introducing (2024)
Answer for the worst case in your scope, not the typical one. Each level carries a tell: the thing that would settle it, so the score is something you checked rather than something you agreed with. At levels 4 and 5 that is usually a command you can run this afternoon. At levels 1 and 2 it is usually a question for two colleagues, because the absence of a measurement is not something anyone can go and look at.
If nobody in your scope can produce it, that is a 1. Not knowing is the level, not a gap in the instrument. If the dimension itself is unfamiliar, read it at length first.
The wording of these levels changed since you answered this one. Your answer is still here. Confirm it or change it.
Choose the level you recognise. There is no skip here: the verdict is the lowest of your scores, and a dimension you never scored cannot be part of it. If you do not know, that is a 1. If the question is unanswerable because your scope is too wide, rather than because your system has no answer, a narrower assessment will tell you more than this one.
Enforcement & Release
Question 17 of 20
Hard guardrails vs cultural agreements
Every organisation has standards. Some are walls: the build fails, the merge is refused, the commit never lands. The rest are agreements, living in a wiki, in review habits and in what people remember on a good day. An agent absorbs none of the second kind, because it never sat in your standup and never read your wiki.
At its best the sort is a live practice, and the agreed column keeps getting shorter because the expensive entries move into the pipeline. At its worst nothing separates the two, so a documented standard and an enforced one feel identical in a meeting, and only one of them still holds during a night-time incident.
Take the standards your teams rely on anywhere in this scope and sort them into two columns: the ones a machine refuses to allow through, and the ones people have agreed to. Do you know how long the second column is?
The research
- Sadowski, Aftandilian, Eagle, Miller-Cushon and Jaspan, Lessons from Building Static Analysis Tools at Google (Communications of the ACM, 2018)
- Gu and Solar-Lezama, Study maps roadblocks to autonomous software engineering (MIT News, 2025)
Answer for the worst case in your scope, not the typical one. Each level carries a tell: the thing that would settle it, so the score is something you checked rather than something you agreed with. At levels 4 and 5 that is usually a command you can run this afternoon. At levels 1 and 2 it is usually a question for two colleagues, because the absence of a measurement is not something anyone can go and look at.
If nobody in your scope can produce it, that is a 1. Not knowing is the level, not a gap in the instrument. If the dimension itself is unfamiliar, read it at length first.
The wording of these levels changed since you answered this one. Your answer is still here. Confirm it or change it.
Choose the level you recognise. There is no skip here: the verdict is the lowest of your scores, and a dimension you never scored cannot be part of it. If you do not know, that is a 1. If the question is unanswerable because your scope is too wide, rather than because your system has no answer, a narrower assessment will tell you more than this one.
Enforcement & Release
Question 18 of 20
Architecture enforcement in CI
Architecture is a set of rules about what may depend on what. For decades a senior person held those rules in review, and it worked because of a ratio nobody had to think about: one human could read what one human could write. That ratio is gone. A clean change with green tests can invert a dependency your design forbids, and nothing refuses it, because the boundary was never in the build.
At its best a scoped check fails the build on a boundary breach, and the rule it enforces is small enough that people keep it rather than switch it off. At its worst the diagram is accurate, the reviewers are diligent, and the structure erodes one reasonable commit at a time.
Does the build fail on an architectural violation anywhere in your scope, or do you rely on a reviewer noticing the one line that inverted a dependency?
The research
- Sas and Avgeriou, On the Evolution and Impact of Architectural Smells: An Industrial Case Study (Empirical Software Engineering, 2022)
- Slater, Quantitative Analysis of Technical Debt and Pattern Violation in LLM Architectures (2025)
Answer for the worst case in your scope, not the typical one. Each level carries a tell: the thing that would settle it, so the score is something you checked rather than something you agreed with. At levels 4 and 5 that is usually a command you can run this afternoon. At levels 1 and 2 it is usually a question for two colleagues, because the absence of a measurement is not something anyone can go and look at.
If nobody in your scope can produce it, that is a 1. Not knowing is the level, not a gap in the instrument. If the dimension itself is unfamiliar, read it at length first.
The wording of these levels changed since you answered this one. Your answer is still here. Confirm it or change it.
Choose the level you recognise. There is no skip here: the verdict is the lowest of your scores, and a dimension you never scored cannot be part of it. If you do not know, that is a 1. If the question is unanswerable because your scope is too wide, rather than because your system has no answer, a narrower assessment will tell you more than this one.
Enforcement & Release
Question 19 of 20
Dependency hygiene
Somebody used to choose every import. They weighed the licence, the maintenance, whether the project was alive. That habit made an import line boring, and reviewers learned to read it as plumbing. Now the import line is written by a model working from a frozen snapshot of the world, which sometimes names a package that does not exist and can be registered by somebody else tomorrow.
At its best imports are checked against the live registry and gated before they land, with a person accountable for the decision. At its worst the review reads the forty lines of logic carefully and approves without reading the one line that added an unknown package to production.
Of the dependencies running in production anywhere in your scope, how many did a human actually choose?
The research
- Spracklen, Wijewickrama, Sakib, Maiti, Viswanath and Jadliwala, We Have a Package for You! A Comprehensive Analysis of Package Hallucinations by Code Generating LLMs (USENIX Security Symposium, 2025)
- Rahman, Paramitha, Enck and Williams, How Quickly Do Development Teams Update Their Vulnerable Dependencies? (2024)
Answer for the worst case in your scope, not the typical one. Each level carries a tell: the thing that would settle it, so the score is something you checked rather than something you agreed with. At levels 4 and 5 that is usually a command you can run this afternoon. At levels 1 and 2 it is usually a question for two colleagues, because the absence of a measurement is not something anyone can go and look at.
If nobody in your scope can produce it, that is a 1. Not knowing is the level, not a gap in the instrument. If the dimension itself is unfamiliar, read it at length first.
The wording of these levels changed since you answered this one. Your answer is still here. Confirm it or change it.
Choose the level you recognise. There is no skip here: the verdict is the lowest of your scores, and a dimension you never scored cannot be part of it. If you do not know, that is a 1. If the question is unanswerable because your scope is too wide, rather than because your system has no answer, a narrower assessment will tell you more than this one.
Enforcement & Release
Question 20 of 20
Release discipline
A release is where a change stops being yours and starts being someone else's problem. Your consumers depend on what they observed, not on what you documented, so a change that breaks nothing in your contract can still break them. Code generation raised how much you publish and left everyone downstream reading at the rate they always did.
At its best the visibility is engineered into the runtime: deprecation signals, alerts on a changed signature, notice that reaches consumers ahead of the change. At its worst the release note is accurate, clinical and useless, and the first person to notice a break is a customer.
When something you released last month changed behaviour, how did the people depending on it find out? And would that still work at the rate your scope now ships?
The research
- Lehman, Programs, Life Cycles, and Laws of Software Evolution (Proceedings of the IEEE, 1980)
- Wang, Huang, Zhang, Feng, Zhang, Liu and Peng, LLMs Meet Library Evolution: Evaluating Deprecated API Usage in LLM-Based Code Completion (ICSE 2025)
Answer for the worst case in your scope, not the typical one. Each level carries a tell: the thing that would settle it, so the score is something you checked rather than something you agreed with. At levels 4 and 5 that is usually a command you can run this afternoon. At levels 1 and 2 it is usually a question for two colleagues, because the absence of a measurement is not something anyone can go and look at.
If nobody in your scope can produce it, that is a 1. Not knowing is the level, not a gap in the instrument. If the dimension itself is unfamiliar, read it at length first.
The wording of these levels changed since you answered this one. Your answer is still here. Confirm it or change it.
Choose the level you recognise. There is no skip here: the verdict is the lowest of your scores, and a dimension you never scored cannot be part of it. If you do not know, that is a 1. If the question is unanswerable because your scope is too wide, rather than because your system has no answer, a narrower assessment will tell you more than this one.
Done
You have already scored all twenty.
Every dimension is answered and your report is waiting. You can read it again, or change an answer: nothing here is fixed, and a corrected answer gives you a corrected report.