You left one unfinished
You have scored 0 of 20 on this device, and nothing expired while you were away. Answers here never do.
Starting fresh keeps the old attempt on this device rather than deleting it.
Before question one
What are you scoring?
A score with no scope is a number without units, so this is the one thing to settle before question one. It is not an administrative step: it decides what the number means.
Widening the scope should lower your score, because the worst case gets worse. Across a business unit with a modernisation half done, that usually means a floor of 1 on most of the twenty. True, and close to useless.
If you are halfway through a modernisation, do not do one assessment. Do two. Score the platform you have built. Then score the estate it has not replaced yet. The distance between those two numbers is where your readiness boundary actually lies, and it is worth more than either number on its own.
It appears on your report so you know which system you scored. It stays on this device and is never transmitted, because it is the one thing you could type here that would identify your organisation.
Two rules before you start. If you do not know the answer to a question, that is a 1. Not knowing is the level, not a gap in the instrument.
And if you are halfway through a modernisation, you will recognise your organisation in two levels at once: the new platform on one, the estate it has not replaced yet on another. Both are true. Score the worse one, because that is the one an agent reaches first.
Foundation & Structure
Question 1 of 20
Repository structure
Your repository setup is the map an agent works from: how many repositories there are, where the boundaries between them run, and whether anything except somebody's memory keeps work on the right side of them. It has no picture of your estate other than the one it can read. So the question is whether that setup helps it reason about the system, or forces it to guess.
At its best the machine holds the shape, whichever model you run: scoped builds and enforced module boundaries in a monorepo, versioned and enforced contracts across many repositories, so crossing a boundary fails rather than merely being noticed. At its worst the structure accumulated rather than being chosen, is written down and nothing more, and holds only for as long as the right person reads the pull request.
The research
- Luo et al., RepoCod: Can Language Models Replace Programmers? (ACL 2025)
- Zhang et al., LLM Hallucinations in Practical Code Generation (ACM PACMSE 2025)
Answer for the worst case in your scope, not the typical one. Each level carries a tell: the thing that would settle it, so the score is something you checked rather than something you agreed with. At levels 4 and 5 that is usually a command you can run this afternoon. At levels 1 and 2 it is usually a question for two colleagues, because the absence of a measurement is not something anyone can go and look at.
The wording of these levels changed since you answered this one. Your answer is still here. Confirm it or change it.
Choose the level you recognise. There is no skip here: the verdict is the lowest of your scores, and a dimension you never scored cannot be part of it. If you do not know, that is a 1. If the question is unanswerable because your scope is too wide, rather than because your system has no answer, a narrower assessment will tell you more than this one.
Foundation & Structure
Question 2 of 20
Cognitive load per module
This is how much of your system somebody has to read in order to change one thing. A module that needs five others read first costs a new engineer a week, and costs an agent its whole context window every session, because unlike your engineers it never builds the familiarity that makes a tangled module survivable. So the question is whether anything in your system knows how tangled the worst one is, or only your people do.
At its best coupling and complexity carry a budget the build enforces, and that budget survives the release that wants to breach it. At its worst there is no number, and the only signal is which parts of the system people quietly avoid.
The research
- He, Miller, Agarwal, Kaestner and Vasilescu, Speed at the Cost of Quality: How Cursor AI Increases Complexity (MSR 2026)
- Rando et al., LongCodeBench: Evaluating Coding LLMs at 1M Context Windows (2025)
Answer for the worst case in your scope, not the typical one. Each level carries a tell: the thing that would settle it, so the score is something you checked rather than something you agreed with. At levels 4 and 5 that is usually a command you can run this afternoon. At levels 1 and 2 it is usually a question for two colleagues, because the absence of a measurement is not something anyone can go and look at.
The wording of these levels changed since you answered this one. Your answer is still here. Confirm it or change it.
Choose the level you recognise. There is no skip here: the verdict is the lowest of your scores, and a dimension you never scored cannot be part of it. If you do not know, that is a 1. If the question is unanswerable because your scope is too wide, rather than because your system has no answer, a narrower assessment will tell you more than this one.
Foundation & Structure
Question 3 of 20
Domain language consistency
This is whether one concept has one name. An agent learns your vocabulary from your code and from nowhere else, so when four services call the same thing four different things, it learns all four as equally correct and then produces a fifth. So the question is what three teams anywhere in your scope would say if you asked each of them to define your most contested domain word.
At its best the glossary is machine-checked: a banned synonym fails a lint rule, and a new public name is checked against the vocabulary before it merges. At its worst there is an accurate glossary that took real work, sitting on a wiki page nothing in the pipeline has ever opened.
The research
- Fakhoury et al., Cognitive Load from Linguistic Antipatterns, measured with fNIRS brain imaging (ICPC 2018, Distinguished Paper)
- Aghajani et al., Linguistic Antipatterns in APIs (2018)
Answer for the worst case in your scope, not the typical one. Each level carries a tell: the thing that would settle it, so the score is something you checked rather than something you agreed with. At levels 4 and 5 that is usually a command you can run this afternoon. At levels 1 and 2 it is usually a question for two colleagues, because the absence of a measurement is not something anyone can go and look at.
The wording of these levels changed since you answered this one. Your answer is still here. Confirm it or change it.
Choose the level you recognise. There is no skip here: the verdict is the lowest of your scores, and a dimension you never scored cannot be part of it. If you do not know, that is a 1. If the question is unanswerable because your scope is too wide, rather than because your system has no answer, a narrower assessment will tell you more than this one.
Foundation & Structure
Question 4 of 20
Semantic drift
Names fragment under change, in every system, always. Refactoring is the work that used to slow that down, and it is the first thing dropped when a deadline approaches. This one is about the rate rather than the state, so the question is whether the same concept has one name across your scope or several, and whether you know if that is worse than a year ago.
At its best drift is tracked over time and the trend is what gets consolidation scheduled as delivery rather than requested as a favour. At its worst nobody could say whether the system is more coherent than it was a year ago or less, so the question gets settled by whoever sounds most certain.
The research
- Lehman, Laws of Software Evolution (IEEE 1980)
- Wang et al., Deep Learning-Based Identification of Inconsistent Method Names (Empirical Software Engineering, 2025)
Answer for the worst case in your scope, not the typical one. Each level carries a tell: the thing that would settle it, so the score is something you checked rather than something you agreed with. At levels 4 and 5 that is usually a command you can run this afternoon. At levels 1 and 2 it is usually a question for two colleagues, because the absence of a measurement is not something anyone can go and look at.
The wording of these levels changed since you answered this one. Your answer is still here. Confirm it or change it.
Choose the level you recognise. There is no skip here: the verdict is the lowest of your scores, and a dimension you never scored cannot be part of it. If you do not know, that is a 1. If the question is unanswerable because your scope is too wide, rather than because your system has no answer, a narrower assessment will tell you more than this one.
Truth Signals
Question 5 of 20
CI reliability
A pipeline is a safety net only while people believe it. When red sometimes means broken and sometimes means unlucky, teams learn to press rerun, and they learn it long before AI arrives. Then the volume of changes goes up and that habit is the last thing standing between generated code and production. So the question is whether red reliably means broken here, or whether everyone has quietly agreed to avoid the question.
At its best the flake rate is measured, driven down and enforced, so a red build is treated as information. At its worst nobody knows the flake rate, reruns are uncounted, and green gets rerun too, because the signal is not trusted in either direction.
The research
- Aïdasso, Bordeleau and Tizghadam, On the Illusion of Success: An Empirical Study of Build Reruns and Silent Failures in Industrial CI (2024)
- Gruber and Fraser, A Survey on How Test Flakiness Affects Developers and What Support They Need To Address It (ICST 2022)
Answer for the worst case in your scope, not the typical one. Each level carries a tell: the thing that would settle it, so the score is something you checked rather than something you agreed with. At levels 4 and 5 that is usually a command you can run this afternoon. At levels 1 and 2 it is usually a question for two colleagues, because the absence of a measurement is not something anyone can go and look at.
The wording of these levels changed since you answered this one. Your answer is still here. Confirm it or change it.
Choose the level you recognise. There is no skip here: the verdict is the lowest of your scores, and a dimension you never scored cannot be part of it. If you do not know, that is a 1. If the question is unanswerable because your scope is too wide, rather than because your system has no answer, a narrower assessment will tell you more than this one.
Truth Signals
Question 6 of 20
Test meaningfulness
A test does two things: it runs the code, and it says what the answer should have been. Coverage measures the first. Protection comes from the second, and that is the half a model cannot supply, because it can read what your code does and not what it was meant to do. So the question is whether your suite would notice a change in behaviour, or only notice the code failing to run at all.
At its best something mechanical checks whether the tests catch faults, and the suite is judged on that rather than on how much code it touched. At its worst coverage is high, generated tests assert whatever the implementation happens to return, and the dashboard stays green over a regression.
The research
- Inozemtseva and Holmes, Coverage Is Not Strongly Correlated with Test Suite Effectiveness (ICSE 2014)
- Konstantinou, Degiovanni and Papadakis, Do LLMs Generate Test Oracles That Capture the Actual or the Expected Program Behaviour? (2024)
Answer for the worst case in your scope, not the typical one. Each level carries a tell: the thing that would settle it, so the score is something you checked rather than something you agreed with. At levels 4 and 5 that is usually a command you can run this afternoon. At levels 1 and 2 it is usually a question for two colleagues, because the absence of a measurement is not something anyone can go and look at.
The wording of these levels changed since you answered this one. Your answer is still here. Confirm it or change it.
Choose the level you recognise. There is no skip here: the verdict is the lowest of your scores, and a dimension you never scored cannot be part of it. If you do not know, that is a 1. If the question is unanswerable because your scope is too wide, rather than because your system has no answer, a narrower assessment will tell you more than this one.
Truth Signals
Question 7 of 20
Change absorption
Absorption is what happens after the merge: whether the change stays, whether a reviewer understood it, whether anyone outside engineering found out it happened. AI raises the amount of change arriving and leaves the capacity to absorb it exactly where it was, so the share of work that has to be done again is the first thing to rise. The question is whether you measure that share, and whether you would notice it climbing.
At its best churn and revert rates are tracked on a cadence, and the number is allowed to stop a plan. At its worst code goes in, some of it quietly comes back out, and the effort spent on the second part is invisible to everybody deciding how much more to generate.
The research
- Nagappan and Ball, Use of Relative Code Churn Measures to Predict System Defect Density (ICSE 2005)
- Cohen and Levinthal, Absorptive Capacity: A New Perspective on Learning and Innovation (Administrative Science Quarterly, 1990)
Answer for the worst case in your scope, not the typical one. Each level carries a tell: the thing that would settle it, so the score is something you checked rather than something you agreed with. At levels 4 and 5 that is usually a command you can run this afternoon. At levels 1 and 2 it is usually a question for two colleagues, because the absence of a measurement is not something anyone can go and look at.
The wording of these levels changed since you answered this one. Your answer is still here. Confirm it or change it.
Choose the level you recognise. There is no skip here: the verdict is the lowest of your scores, and a dimension you never scored cannot be part of it. If you do not know, that is a 1. If the question is unanswerable because your scope is too wide, rather than because your system has no answer, a narrower assessment will tell you more than this one.
Truth Signals
Question 8 of 20
Technical debt
Technical debt is the work you already owe on the system you already have. AI does not pay it down. It draws on that debt faster, because generated code lands in the same tangles and the refactor that would have cleaned them up is the first thing dropped under delivery pressure. So the question is whether the debt is a number your business case carries, or a belief that the speed will fund the cleanup later.
At its best the debt is quantified in money, and that figure sits inside the plan for the acceleration rather than beside it. At its worst it is an engineering complaint, real to everyone who touches the code and absent from every document where the money is decided.
The research
- METR, Measuring the Impact of Early-2025 AI on Experienced Open-Source Developer Productivity (2025)
- GitClear, AI Copilot Code Quality: 2025 Data Suggests 4x Growth in Code Clones (2025)
Answer for the worst case in your scope, not the typical one. Each level carries a tell: the thing that would settle it, so the score is something you checked rather than something you agreed with. At levels 4 and 5 that is usually a command you can run this afternoon. At levels 1 and 2 it is usually a question for two colleagues, because the absence of a measurement is not something anyone can go and look at.
The wording of these levels changed since you answered this one. Your answer is still here. Confirm it or change it.
Choose the level you recognise. There is no skip here: the verdict is the lowest of your scores, and a dimension you never scored cannot be part of it. If you do not know, that is a 1. If the question is unanswerable because your scope is too wide, rather than because your system has no answer, a narrower assessment will tell you more than this one.
Flow & Throughput
Question 9 of 20
Trunk-based development
Every day a branch stays open, the distance between it and the trunk grows, and the cost of closing that distance grows with it. AI raised how much code changes in a day and left most branching strategies exactly where they were, so the merge that used to be fine is now a two-day job with real logic in it. The question is whether branch lifetime is bounded by something, or by everybody's good intentions.
At its best branches live hours, because feature flags and fast tests make that possible and a rule makes it required. At its worst nobody knows the oldest branch, integration is an event near a deadline, and the conflicts arrive long after the person who wrote the code stopped remembering it.
The research
- Kasi and Sarma, An Empirical Investigation into Merge Conflicts and Their Effect on Software Quality (Empirical Software Engineering, 2013)
- Ghiotto, Murta, Barros and van der Hoek, On the Nature of Merge Conflicts: A Study of 2,731 Open Source Java Projects (IEEE Transactions on Software Engineering, 2018)
Answer for the worst case in your scope, not the typical one. Each level carries a tell: the thing that would settle it, so the score is something you checked rather than something you agreed with. At levels 4 and 5 that is usually a command you can run this afternoon. At levels 1 and 2 it is usually a question for two colleagues, because the absence of a measurement is not something anyone can go and look at.
The wording of these levels changed since you answered this one. Your answer is still here. Confirm it or change it.
Choose the level you recognise. There is no skip here: the verdict is the lowest of your scores, and a dimension you never scored cannot be part of it. If you do not know, that is a 1. If the question is unanswerable because your scope is too wide, rather than because your system has no answer, a narrower assessment will tell you more than this one.
Flow & Throughput
Question 10 of 20
Velocity and the constraint
Your delivery pipeline is a chain of stages and one of them has the least capacity. Double what arrives and that stage saturates first. Relieve it and the limit moves to the next. Waiting does not grow in proportion to load either: once a stage is busy, a modest increase produces a queue out of all proportion to it. So the question is whether you know which stage yours is, or whether you will meet it during an incident.
At its best the constraint is named, work in progress is capped by tooling that refuses new work past the line, and the cap moves when the constraint does. At its worst every stage looks busy, nobody can say which one is the limit, and capacity gets added where it is easiest rather than where it would help.
The research
- Reinertsen, The Principles of Product Development Flow (book, 2009)
Answer for the worst case in your scope, not the typical one. Each level carries a tell: the thing that would settle it, so the score is something you checked rather than something you agreed with. At levels 4 and 5 that is usually a command you can run this afternoon. At levels 1 and 2 it is usually a question for two colleagues, because the absence of a measurement is not something anyone can go and look at.
The wording of these levels changed since you answered this one. Your answer is still here. Confirm it or change it.
Choose the level you recognise. There is no skip here: the verdict is the lowest of your scores, and a dimension you never scored cannot be part of it. If you do not know, that is a 1. If the question is unanswerable because your scope is too wide, rather than because your system has no answer, a narrower assessment will tell you more than this one.
Flow & Throughput
Question 11 of 20
Organizational design
Your org chart decides which changes are cheap and which need three handoffs. AI raises what each team produces on its own, and it does nothing at all to the seams between them, so a feature crossing four teams gets faster inside each team and no faster between them. The question is whether anything in your organisation watches those seams, or whether the queues between teams are only ever discovered as a date that slipped.
At its best the boundaries are watched with numbers, they move when the numbers say so, and anyone who understands the domain can review a change inside it. At its worst ownership is fixed, every crossing waits for the one team allowed to touch that component, and the structure is revisited only in a reorganisation.
The research
- MacCormack, Baldwin and Rusnak, Exploring the Duality between Product and Organizational Architectures (Research Policy, 2012)
- Nagappan, Murphy and Basili, The Influence of Organizational Structure on Software Quality (ICSE 2008)
Answer for the worst case in your scope, not the typical one. Each level carries a tell: the thing that would settle it, so the score is something you checked rather than something you agreed with. At levels 4 and 5 that is usually a command you can run this afternoon. At levels 1 and 2 it is usually a question for two colleagues, because the absence of a measurement is not something anyone can go and look at.
The wording of these levels changed since you answered this one. Your answer is still here. Confirm it or change it.
Choose the level you recognise. There is no skip here: the verdict is the lowest of your scores, and a dimension you never scored cannot be part of it. If you do not know, that is a 1. If the question is unanswerable because your scope is too wide, rather than because your system has no answer, a narrower assessment will tell you more than this one.
Flow & Throughput
Question 12 of 20
Code ownership and review routing
Review is where generated code meets human judgement, and routing decides whose. When every change in an area goes to the one person who owns it, tripling the volume does not triple that person. It lengthens a queue, and then it lowers the standard of what comes out of the queue. The question is how many people could competently review a change to your most critical area, and whether the routing knows that or only knows one name.
At its best routing spreads review across people who genuinely know the area, and queue depth is watched, so a growing wait is caught before review becomes approval without reading. At its worst one name sits in a file, the queue behind it is invisible, and nobody learns how narrow it was until that person is suddenly unavailable.
The research
- Bird, Nagappan, Murphy, Gall and Devanbu, Don't Touch My Code! Examining the Effects of Ownership on Software Quality (ESEC/FSE 2011)
- Thongtanunam, McIntosh, Hassan and Iida, Revisiting Code Ownership and Its Relationship with Software Quality in the Scope of Modern Code Review (ICSE 2016)
Answer for the worst case in your scope, not the typical one. Each level carries a tell: the thing that would settle it, so the score is something you checked rather than something you agreed with. At levels 4 and 5 that is usually a command you can run this afternoon. At levels 1 and 2 it is usually a question for two colleagues, because the absence of a measurement is not something anyone can go and look at.
The wording of these levels changed since you answered this one. Your answer is still here. Confirm it or change it.
Choose the level you recognise. There is no skip here: the verdict is the lowest of your scores, and a dimension you never scored cannot be part of it. If you do not know, that is a 1. If the question is unanswerable because your scope is too wide, rather than because your system has no answer, a narrower assessment will tell you more than this one.
Memory & Intent
Question 13 of 20
Decision memory (ADRs)
An architectural decision is two things: the shape of the system, and the reason it has that shape. Only the first survives in the code. An agent asked to add a feature takes the cheapest route, straight across the boundary you drew, and it has no way to know the boundary was expensive, because the reason lived in a meeting. So the question is whether a decision from two years ago is findable where the work happens, or only inside whoever made it.
At its best decisions sit in the repository beside the code, dated, superseded when they change, and a reviewer can point at one. At its worst they exist as slides, a chat thread and one person's memory, and the first change that finds them inconvenient quietly reverses them.
The research
- Robillard et al., Turnover-Induced Knowledge Loss in Practice (ESEC/FSE 2021)
- Storey, From Technical Debt to Cognitive and Intent Debt (2026)
Answer for the worst case in your scope, not the typical one. Each level carries a tell: the thing that would settle it, so the score is something you checked rather than something you agreed with. At levels 4 and 5 that is usually a command you can run this afternoon. At levels 1 and 2 it is usually a question for two colleagues, because the absence of a measurement is not something anyone can go and look at.
The wording of these levels changed since you answered this one. Your answer is still here. Confirm it or change it.
Choose the level you recognise. There is no skip here: the verdict is the lowest of your scores, and a dimension you never scored cannot be part of it. If you do not know, that is a 1. If the question is unanswerable because your scope is too wide, rather than because your system has no answer, a narrower assessment will tell you more than this one.
Memory & Intent
Question 14 of 20
Requirements quality
This is whether the brief an agent picks up says enough to build the right thing. An agent does not ask clarifying questions the way a developer does: it fills the gap with the most likely reading and writes confident, plausible code against it. The vaguer the item, the more confidently wrong the output, and the faster it arrives. So the question is whether the top item of any backlog in your scope could be picked up and built without guessing.
At its best an item cannot be pulled until it is a testable spec, and the check runs before the work starts rather than at review. At its worst items are a title and a sentence of context, and what finished means lives in a conversation that has not happened yet.
The research
- Wu et al., ClarifyCoder: Clarification-Aware Fine-Tuning for Programmatic Problem Solving (2025)
- Li, Wu and Chang, ClarEval: A Benchmark for Evaluating Clarification Skills of Code Agents (2026)
Answer for the worst case in your scope, not the typical one. Each level carries a tell: the thing that would settle it, so the score is something you checked rather than something you agreed with. At levels 4 and 5 that is usually a command you can run this afternoon. At levels 1 and 2 it is usually a question for two colleagues, because the absence of a measurement is not something anyone can go and look at.
The wording of these levels changed since you answered this one. Your answer is still here. Confirm it or change it.
Choose the level you recognise. There is no skip here: the verdict is the lowest of your scores, and a dimension you never scored cannot be part of it. If you do not know, that is a 1. If the question is unanswerable because your scope is too wide, rather than because your system has no answer, a narrower assessment will tell you more than this one.
Memory & Intent
Question 15 of 20
Security assumptions
Most security lives in people. The engineer who knows tokens never go in local storage, the architect who remembers why this service signs its own requests. A model has no onboarding and no corridor conversations. It reads your code, and if three patterns for the same thing are in there, one secure and two historical, it treats all three as examples. So the question is where an agent would learn the rules your team holds, and what happens if the honest answer is nowhere.
At its best the assumption exists once, as a canonical pattern with a scan that fails the build when something deviates. At its worst it is genuinely believed, genuinely important, and written nowhere a machine could find it.
The research
- Perry, Srivastava, Kumar and Boneh, Do Users Write More Insecure Code with AI Assistants? (ACM CCS 2023)
- Pearce, Ahmad, Tan, Dolan-Gavitt and Karri, Asleep at the Keyboard? Assessing the Security of GitHub Copilot's Code Contributions (IEEE Symposium on Security and Privacy, 2022)
Answer for the worst case in your scope, not the typical one. Each level carries a tell: the thing that would settle it, so the score is something you checked rather than something you agreed with. At levels 4 and 5 that is usually a command you can run this afternoon. At levels 1 and 2 it is usually a question for two colleagues, because the absence of a measurement is not something anyone can go and look at.
The wording of these levels changed since you answered this one. Your answer is still here. Confirm it or change it.
Choose the level you recognise. There is no skip here: the verdict is the lowest of your scores, and a dimension you never scored cannot be part of it. If you do not know, that is a 1. If the question is unanswerable because your scope is too wide, rather than because your system has no answer, a narrower assessment will tell you more than this one.
Memory & Intent
Question 16 of 20
Documentation integrity
Documentation used to have a repair mechanism built into its reader. A person who hit a step that no longer worked asked somebody, and the asking quietly held the whole thing together. An agent cannot ask. It reads your instructions as ground truth and generates confidently against a description that stopped matching reality in March. So the question is whether anything checks your setup and run instructions against the running system, or whether the checking was always done by people who could simply go and ask somebody.
At its best the load-bearing instructions are executed in the pipeline from a clean checkout, so drift turns the build red. At its worst the README is plausible, confident and four months stale, and nothing in the organisation would notice.
The research
- Ashik, Wang, Chen, Asaduzzaman and Tian, When LLMs Lag Behind: Knowledge Conflicts from Evolving APIs in Code Generation (2026)
- Radmanesh, Imani, Ahmed and Moshirpour, Investigating the Impact of Code Comment Inconsistency on Bug Introducing (2024)
Answer for the worst case in your scope, not the typical one. Each level carries a tell: the thing that would settle it, so the score is something you checked rather than something you agreed with. At levels 4 and 5 that is usually a command you can run this afternoon. At levels 1 and 2 it is usually a question for two colleagues, because the absence of a measurement is not something anyone can go and look at.
The wording of these levels changed since you answered this one. Your answer is still here. Confirm it or change it.
Choose the level you recognise. There is no skip here: the verdict is the lowest of your scores, and a dimension you never scored cannot be part of it. If you do not know, that is a 1. If the question is unanswerable because your scope is too wide, rather than because your system has no answer, a narrower assessment will tell you more than this one.
Enforcement & Release
Question 17 of 20
Hard guardrails vs cultural agreements
Every organisation has standards. Some are walls: the build fails, the merge is refused, the commit never lands. The rest are agreements, living in a wiki, in review habits and in what people remember on a good day. An agent absorbs none of the second kind, because it never sat in your standup and never read your wiki. So the question is whether you know which of your standards are walls, and how long the other list is.
At its best the sort is a live practice, and the agreed column keeps getting shorter because the expensive entries move into the pipeline. At its worst nothing separates the two, so a documented standard and an enforced one feel identical in a meeting, and only one of them still holds during a night-time incident.
The research
- Sadowski, Aftandilian, Eagle, Miller-Cushon and Jaspan, Lessons from Building Static Analysis Tools at Google (Communications of the ACM, 2018)
- Gu and Solar-Lezama, Study maps roadblocks to autonomous software engineering (MIT News, 2025)
Answer for the worst case in your scope, not the typical one. Each level carries a tell: the thing that would settle it, so the score is something you checked rather than something you agreed with. At levels 4 and 5 that is usually a command you can run this afternoon. At levels 1 and 2 it is usually a question for two colleagues, because the absence of a measurement is not something anyone can go and look at.
The wording of these levels changed since you answered this one. Your answer is still here. Confirm it or change it.
Choose the level you recognise. There is no skip here: the verdict is the lowest of your scores, and a dimension you never scored cannot be part of it. If you do not know, that is a 1. If the question is unanswerable because your scope is too wide, rather than because your system has no answer, a narrower assessment will tell you more than this one.
Enforcement & Release
Question 18 of 20
Architecture enforcement in CI
Architecture is a set of rules about what may depend on what. For decades a senior person held those rules in review, and it worked because of a ratio nobody had to think about: one human could read what one human could write. That ratio is gone. A clean change with green tests can invert a dependency your design forbids, and nothing refuses it, because the boundary was never in the build. So the question is whether your rules are executable or remembered.
At its best a scoped check fails the build on a boundary breach, and the rule it enforces is small enough that people keep it rather than switch it off. At its worst the diagram is accurate, the reviewers are diligent, and the structure erodes one reasonable commit at a time.
The research
- Sas and Avgeriou, On the Evolution and Impact of Architectural Smells: An Industrial Case Study (Empirical Software Engineering, 2022)
- Slater, Quantitative Analysis of Technical Debt and Pattern Violation in LLM Architectures (2025)
Answer for the worst case in your scope, not the typical one. Each level carries a tell: the thing that would settle it, so the score is something you checked rather than something you agreed with. At levels 4 and 5 that is usually a command you can run this afternoon. At levels 1 and 2 it is usually a question for two colleagues, because the absence of a measurement is not something anyone can go and look at.
The wording of these levels changed since you answered this one. Your answer is still here. Confirm it or change it.
Choose the level you recognise. There is no skip here: the verdict is the lowest of your scores, and a dimension you never scored cannot be part of it. If you do not know, that is a 1. If the question is unanswerable because your scope is too wide, rather than because your system has no answer, a narrower assessment will tell you more than this one.
Enforcement & Release
Question 19 of 20
Dependency hygiene
Somebody used to choose every import. They weighed the licence, the maintenance, whether the project was alive. That habit made an import line boring, and reviewers learned to read it as plumbing. Now the import line is written by a model working from a frozen snapshot of the world, which sometimes names a package that does not exist and can be registered by somebody else tomorrow. So the question is who chooses now, and what happens at the moment the choice is made.
At its best imports are checked against the live registry and gated before they land, with a person accountable for the decision. At its worst the review reads the forty lines of logic carefully and approves without reading the one line that added an unknown package to production.
The research
- Spracklen, Wijewickrama, Sakib, Maiti, Viswanath and Jadliwala, We Have a Package for You! A Comprehensive Analysis of Package Hallucinations by Code Generating LLMs (USENIX Security Symposium, 2025)
- Rahman, Paramitha, Enck and Williams, How Quickly Do Development Teams Update Their Vulnerable Dependencies? (2024)
Answer for the worst case in your scope, not the typical one. Each level carries a tell: the thing that would settle it, so the score is something you checked rather than something you agreed with. At levels 4 and 5 that is usually a command you can run this afternoon. At levels 1 and 2 it is usually a question for two colleagues, because the absence of a measurement is not something anyone can go and look at.
The wording of these levels changed since you answered this one. Your answer is still here. Confirm it or change it.
Choose the level you recognise. There is no skip here: the verdict is the lowest of your scores, and a dimension you never scored cannot be part of it. If you do not know, that is a 1. If the question is unanswerable because your scope is too wide, rather than because your system has no answer, a narrower assessment will tell you more than this one.
Enforcement & Release
Question 20 of 20
Release discipline
A release is where a change stops being yours and starts being someone else's problem. Your consumers depend on what they observed, not on what you documented, so a change that breaks nothing in your contract can still break them. Code generation raised how much you publish and left everyone downstream reading at the rate they always did. The question is whether anything tells them what changed before it breaks something in production.
At its best the visibility is engineered into the runtime: deprecation signals, alerts on a changed signature, notice that reaches consumers ahead of the change. At its worst the release note is accurate, clinical and useless, and the first person to notice a break is a customer.
The research
- Lehman, Programs, Life Cycles, and Laws of Software Evolution (Proceedings of the IEEE, 1980)
- Wang, Huang, Zhang, Feng, Zhang, Liu and Peng, LLMs Meet Library Evolution: Evaluating Deprecated API Usage in LLM-Based Code Completion (ICSE 2025)
Answer for the worst case in your scope, not the typical one. Each level carries a tell: the thing that would settle it, so the score is something you checked rather than something you agreed with. At levels 4 and 5 that is usually a command you can run this afternoon. At levels 1 and 2 it is usually a question for two colleagues, because the absence of a measurement is not something anyone can go and look at.
The wording of these levels changed since you answered this one. Your answer is still here. Confirm it or change it.
Choose the level you recognise. There is no skip here: the verdict is the lowest of your scores, and a dimension you never scored cannot be part of it. If you do not know, that is a 1. If the question is unanswerable because your scope is too wide, rather than because your system has no answer, a narrower assessment will tell you more than this one.
Done
You have already scored all twenty.
Every dimension is answered and your report is waiting. You can read it again, or change an answer: nothing here was submitted anywhere, so changing your mind costs nothing.