What to stop measuring, what to start measuring, and what to tell your board when they ask
I have argued in two prior pieces that the binding constraint in software delivery has moved from writing code to verifying it, and that the org chart should be restructured around that constraint. Both arguments contain the same unstated assumption: that you can see what is actually happening in your organization. Almost nobody can.
DX’s research across more than 180 companies puts weekly AI adoption at roughly 93% of developers, and finds that only about 10% of organizations can produce a defensible productivity number. That gap is not a rounding error. It means every board conversation about AI ROI right now is running on sentiment, and every reorg justified by AI adoption is being defended with self-reported data that METR has demonstrated is confidently wrong in the wrong direction.
If your instrumentation is broken, everything downstream of it is decorative. So this piece is about the instruments. What to stop reaching for, what to build instead, and what to say when your CEO asks the question you have been avoiding.
The instruments most leaders currently trust are broken
Before proposing anything to add, I want to be specific about what to stop using. Each of the following is in wide use, and each fails in a way that is not obvious until you look for it.
Surveys, on their own. Self-report is the default instrument because it is cheap and produces a number quickly. It is also the most misleading instrument available on this specific question. METR’s randomized trial found experienced developers were 19% slower on tasks in familiar repositories while believing they had been 20% faster. That is not a small measurement error. It is a directional inversion. Across the wider industry, self-reported gains of 25% or more are common while measured typical gains sit at 10 to 15%. If your primary evidence for AI impact is a survey, you have a very high probability of defending a number that telemetry does not support. Surveys have a role, which I will come back to. That role is not the top of the funnel.
Lines of code and pull request counts. These were shaky before agents. They collapsed the moment agents started producing both. Faros AI’s telemetry across 10,000 developers in 1,255 teams found that teams with heavy AI adoption merged 98% more pull requests while PR review time grew 91%, average PR size grew 154%, and organizational DORA metrics showed no measurable improvement. Their 2026 follow-up, across 22,000 developers, found bugs per developer up 54% and the incident-to-PR ratio more than tripled. Rewarding PR count in this environment optimizes for the exact behavior that produces the paradox.
Adoption rate as a proxy for impact. Tool utilization tells you the tool is present, not that it is working. DX’s data shows adoption plateauing around 60 to 70% weekly active use in leading organizations, and high adoption coexisting comfortably with flat or negative delivery impact. DORA published a warning in June 2026 against a practice they named tokenmaxxing, which is tracking and rewarding raw token consumption via internal leaderboards. Do not do this. It gamifies the input, and the input is the one thing you should be trying to spend less of per unit of outcome.
Any one of the DORA four in isolation. The four metrics were designed to be read together and held in tension. AI adoption pulls throughput and stability in opposite directions, which means cherry-picking a single number lets you tell almost any story. A lead time chart showing acceleration is worthless without the matching change failure rate. A stability chart showing no incidents is worthless without deployment rework, because rework absorbs damage that never shows up as a page.
Any metric your vendor’s dashboard emphasizes because it is easy to display. Suggestion acceptance rates, tokens consumed, characters generated. None of these are outcomes. They are what the vendor can see, and they became prominent because they are cheap to instrument, not because they are true.
The general failure mode is the same across all of these: the instrument was built for a world where writing code was expensive and reviewing it was cheap, and it is now being read in a world where the ratio inverted. What made these instruments useful is what makes them dangerous now.
What to measure instead
There is one structural framework worth adopting, and I am going to name it directly rather than pretend I invented it. DX has published a three-layer AI measurement framework built around utilization, impact, and cost, designed to sit alongside the DX Core 4 productivity metrics. It is the most defensible structure I have seen in this space, and it is vendor-agnostic. I am going to use it as the spine, add the specific metrics I care most about at each layer, and be explicit about the ones I would deprioritize.
Layer one: utilization
This is the foundation, and it is where most organizations stop. Do not stop here, but do measure it.
The question this layer answers: are the tools present, and where.
Worth tracking: weekly active users of agent tooling, share of merged code that is agent-authored, tasks delegated to agents, shadow usage of personal accounts on organizational work. That last one matters more than most leaders admit. DX’s Q1 2026 data found significant shadow usage even in organizations with enterprise licensing, which means your official adoption number is understating the surface area of your governance problem.
What to do with these numbers: use them to segment every downstream analysis. A comparison between AI-assisted and non-AI-assisted work is the single most useful cut you can make. Trend lines mean less than segmentation here.
What not to do with these numbers: treat them as evidence of value. High utilization is a precondition for impact, not a proxy for it.
Layer two: impact
This is where the argument gets made. It is also the layer most organizations underinvest in, because it is harder to instrument.
Measure impact against a frozen pre-AI baseline. If you did not capture the baseline, capture it now for your current adoption state and treat the next twelve months as the measurement window. Without a baseline, everything is anecdote.
The impact metrics I would put on a single dashboard, in this order.
First, first-review wait time, segmented by AI-authored and human-authored. This is where the review queue forms and where the productivity paradox lives. LinearB’s 2026 benchmarks across 8.1 million pull requests found AI-generated PRs waiting 4.6 times longer for a first review than human-written ones. If your review wait times are diverging, you have a problem the vendor dashboard will not show you.
Second, the DORA four as a set. Deployment frequency, lead time for changes, change failure rate, and time to restore. Held together, not cherry-picked. If throughput is up and change failure rate is up faster, you are shipping worse code faster and calling it progress.
Third, deployment rework rate. The share of deploys that require a follow-up deploy within a defined window to fix something the original deploy broke or missed. This is the metric that catches the paradox even when the DORA four look acceptable, because it counts the invisible cost of nearly right code.
Fourth, code churn within a two-week window of authorship. GitClear’s 2026 maintainability research across hundreds of millions of changed lines documented duplicated code blocks up 81% from 2023, within-commit copy-paste up 41%, error-masking constructs up 47%, and two-week churn up 15%. Cross-file reuse, one of the few positive signals of a healthy codebase, dropped 35%. These are indicators of maintainability debt accruing silently, and they are exactly the debt agents produce fastest.
Fifth, PR size distribution, especially the 75th percentile. AI-authored PRs are systematically larger. Large PRs get reviewed less carefully. This is not a preference. It is a system dynamic worth watching monthly.
Sixth, and only after the first five are in place, a developer experience index. DX’s DXI is a defensible one, and there are others. This is where surveys earn their place. Not as the top of the funnel, but as a signal about whether the humans in the system are functional. The measure that matters here is stability over time and consistency across teams, not the absolute number.
I would explicitly deprioritize commit frequency, story point velocity, and any measure of individual output. These are gaming magnets, and every senior engineer I know is already suspicious of them.
Layer three: cost
This is the layer most easily reduced to a wrong number.
The wrong number is spend per engineer. It flatters procurement conversations and answers no useful question, because engineers are not the unit of value.
The two right numbers are spend per merged change and spend per resolved incident. Both anchor cost to outcome. Both make it visible when you are spending more to produce more work that requires more work to fix.
Track total AI spend, absolutely. But when you present cost to a business audience, present it as a ratio to something the business cares about. Cost per feature shipped. Cost per customer-visible defect avoided or introduced. Cost per hour saved in production support. The instrument decides the conversation.
One additional cost signal worth adding, which almost no dashboard tracks: verification cost. The engineering time spent reviewing, testing, and debugging agent-authored work, as a share of total engineering time. This is your paradox alarm. When it starts rising faster than throughput, the tool is producing net negative value in your environment and no other number will make that clear.
The discipline that determines whether any of this works
The framework matters less than the discipline. Three things separate organizations that measure AI usefully from organizations that generate quarterly slides no one trusts.
Instrument before you roll out. This is the discipline nobody wants to hear. If you deploy the tool first and start measuring second, you have destroyed your ability to prove anything. You need at least a quarter of pre-adoption data on the same metrics you plan to use to evaluate adoption. If you are already past that point, the second-best move is to instrument now and treat your current state as the new baseline for the next twelve months. There is no third-best move. Trying to reconstruct a baseline retroactively from git history is a comforting exercise that produces indefensible numbers.
Never tie any of these metrics to individual performance evaluation. The moment code volume, PR count, or agent utilization becomes a factor in someone’s review, the number becomes worthless. DX puts this bluntly in their rollout guidance, and I have seen it fail firsthand in every organization that tried. Team-level measurement, for organizational learning, communicated clearly as such. Individual performance evaluation continues to happen through the mechanisms it always did: outcomes, judgment, and the assessment of humans who work alongside the person.
Read metrics in groups, always. A single number is a story. A set of numbers is evidence. Any metric on any of these three layers, presented alone in a board deck, is misleading by construction. Pair throughput with change failure rate. Pair adoption with impact. Pair cost per change with the trend in first-review wait time. If you build the discipline of always presenting metrics in pairs, you eliminate roughly 80% of the reasoning errors that these dashboards produce.
What to tell your board when they ask
They will ask. Some of them are asking now. The question always arrives in one of two forms.
“What’s the ROI on the AI tools we’re paying for?”
The wrong answer is a percentage from a survey. The right answer has three parts.
Part one, the honest picture of where you are on the J-curve. DORA’s 2026 ROI model shows a first-year return around 39% with an eight-month payback for a 500-person organization, with a J-curve dip before the return arrives. If you are in the dip, name it as the dip and explain what comes after. If you are past the dip, show the throughput and stability numbers together, and be specific about what changed.
Part two, the paradox metrics. Verification cost as a share of engineering time. First-review wait time by authorship. Deployment rework rate. These are the numbers that reveal whether the investment is compounding or leaking. Boards respect leaders who volunteer the leaks before being asked about them.
Part three, the honest ranges. Point to industry telemetry, not to your own survey. DX’s longitudinal analysis puts real organizational productivity gains at 5 to 15% typically, with individual time savings around four hours per week. Faros’ data shows organizational DORA metrics flat despite dramatic individual gains in high-adoption environments. Your numbers should sit inside these ranges or you should be able to explain why they do not.
“Are we behind?”
The honest answer is that nobody knows, because the industry is still measuring the wrong things. The useful answer is to reframe. Adoption is not the race. Verification capacity is. Show your ratio of platform and harness investment to total engineering, and show your verification cost trend. If those numbers are healthy, you are ahead of the organizations racing on adoption alone, and you will still be ahead when the market corrects its measurements.
What to do this quarter
If you have none of this today, and most organizations do not, the ninety-day sequence is:
Weeks 1 to 4: Instrument. Add telemetry for agent-authored share of merged code, first-review wait time by authorship, deployment rework rate, and verification cost as a share of engineering time. Do not change any tooling yet. Publish the numbers to leadership as a baseline, explicitly labeled as such.
Weeks 5 to 8: Compare. Segment every metric by AI-authored versus human-authored, by team, and by service. Look for the divergences. Where is review wait time diverging most sharply? Which teams are showing the paradox and which are not? What do the healthy teams have in common? This is where the real learning is, and it is almost always uncomfortable.
Weeks 9 to 12: Decide. Based on the divergences, choose one intervention. Not five. One. Usually it is a platform investment that reduces the verification tax on the teams that are drowning. Publish the intervention, publish the metrics it should move, publish the timeline. Then measure it.
That is not a framework. It is a habit. Habits compound. Frameworks decay.
The three pieces I have written now form a series with the same underlying claim. Verification is the constraint. The org chart should follow the constraint. And the instruments most leaders trust cannot see the constraint at all.
If I had to put the argument on a single line for a CTO reading this on a Monday morning, it would be this.
You are almost certainly measuring inputs and reporting them as outcomes. Your job this quarter is to stop.
Everything else is downstream of that decision.
Sources
- DX. AI Measurement Framework and Q1 2026 AI-Assisted Engineering Report. Utilization, impact, and cost layers; 93% weekly adoption; ~28% of committed code AI-authored; 5 to 15% real productivity gains.
- Laura Tacho, DX. Measuring the Impact of AI on Software Engineering. Pragmatic Engineer, 2025. Findings from 180+ companies.
- Faros AI. AI Productivity Paradox (2025) and Acceleration Whiplash (2026). Telemetry from 10,000 to 22,000 developers on PR throughput, review time, PR size, and bugs.
- LinearB. 2026 Software Engineering Benchmarks Report. 8.1M pull requests, 4,800 organizations.
- METR. Measuring the Impact of Early-2025 AI on Experienced Open-Source Developer Productivity, July 2025, with the February 2026 update.
- GitClear. The Maintainability Gap: 2026 AI Code Quality Research. Duplicated blocks, copy-paste, refactoring, and churn indicators.
- DORA / Google Cloud. The ROI of AI-assisted Software Development, 2026, including the J-curve model and the June 2026 tokenmaxxing warning.
- Stack Overflow 2025 Developer Survey, on developer frustration with almost-right code.