.png)
AI generated 40% more code this quarter. The dashboard is green. Nobody in the room can say how much of that code reached production, how much came back through review, or whether one customer received anything sooner.
Code volume is the easiest number to produce and the least informative one on the page. AI-driven development needs metrics tied to outcomes: delivery speed, software quality, engineering capacity, and the maturity of the process around all three.
Let’s explore them in more detail.
Why lines of code are a terrible AI KPI
Lines of code are easy to count. That is the entire case for them.
A larger codebase can mean more functionality. It can also mean more review hours, more tests to maintain, more technical debt, and more surface area for defects. Every generated line arrives with a maintenance bill attached, and the metric counting the lines does not track the bill.
The distinction gets sharper as AI-assisted engineering settles into the daily engineering workflow.
McKinsey's research gives the outcome-first approach some weight. The firm surveyed nearly 300 senior leaders at publicly traded companies, with 100 respondents assessing impact across four measures. The top quintile reported the following gains.
The separation between stronger and weaker performers goes past tool adoption. McKinsey measured a 15-percentage-point performance gap between top and bottom performers and found the leaders applying AI across more stages of product development and reworking roles, processes, and ways of working around it.
For a CTO, the takeaway fits on one line: an AI dashboard should show what happens after code generation.
The 10-metric AI engineering scorecard
Ten metrics tell a CTO far more than one adoption number. The set has to cover three questions: how quickly value reaches customers, whether software quality holds up under the new pace, and whether the engineering process can carry AI without producing new bottlenecks.
Group 1: Are we delivering value faster?
- Lead time for value
- Deployment frequency
- Pull request cycle time
Group 2: Are we preserving quality?
- Change failure rate
- Defect escape rate
- Mean time to recovery
- AI rework rate
Group 3: Is our engineering process getting stronger?
- Developer time recovered
- AI workflow coverage
- Cost per successful engineering outcome
AWS recommends measuring generative AI across deployment velocity, code quality, operational efficiency, team productivity and satisfaction, and business impact, with baselines captured before implementation so later results have something to sit against.
Each metric below follows four points: what the number measures, how AI affects the result, what distorts the reading, and which companion metric keeps it honest.
1. Lead time for value
Start the clock at an approved piece of work. Stop it when the change serves production traffic. For a CTO, that interval says more about AI development productivity than lines generated or coding hours saved.
DORA measures a narrower version called change lead time, running from the first commit to successful production deployment, and places it alongside deployment frequency, change failure rate, and recovery time as core software delivery performance metrics.
For an executive scorecard, the clock can start earlier, at an approved requirement or product decision, then follow the work through design, implementation, testing, review, deployment, and release.
AI touches almost every stage on that path. Requirements analysis moves faster. Engineers retrieve technical context sooner. Test cases arrive earlier. Review preparation takes less manual work.
Then the arithmetic gets interesting. A team cuts implementation from three days to one. The PR spends three days in review and two more in testing. The coding metric looks excellent. The customer's calendar has not moved.
DORA's 2026 analysis describes the same tradeoff: AI accelerates initial code generation, and the saved creation time reappears in auditing and verification
Watch for: A shorter coding cycle sitting alongside longer review, testing, security, or deployment queues.
Read alongside: Feature adoption, customer satisfaction, or another customer-facing outcome.
2. Delivery flow: deployment frequency and pull request cycle time
These two show how work moves through the pipeline. Deployment frequency counts successful production releases. Pull request cycle time measures the stretch from PR creation through approval and merge.
AI affects both through implementation, test generation, documentation, review support, and release preparation. Faster authoring also creates more work for reviewers, so the two numbers need to be read against each other.
A team's deployment frequency rises 30% after AI adoption. Encouraging. Then PR cycle time rises by 40%. The team ships more often, and every change waits longer for a human to look at it.
Harness surveyed 700 engineering practitioners and managers across five countries for its 2026 research. 89% of engineering leaders reported improved developer productivity after AI deployment. 81% also reported longer code review times, with 28% putting the increase above 30%.
Respondents estimated that roughly 31% of developer time now goes to invisible work: reviewing AI-generated code, fixing bugs, switching between tools.
Watch for: Smaller PRs can raise deployment frequency without improving customer value. Faster approval can also come from a lighter review.
Read alongside: Change failure rate for deployment frequency, review quality, or rework for PR cycle time.
3. Delivery stability: change failure rate and mean time to recovery
Speed needs a stability check.
Change failure rate tracks the share of production changes leading to incidents, rollbacks, remediation work, or service degradation. Mean time to recovery tracks how fast the team restores service afterward.
AI helps on both sides of the incident. Before release, it generates test cases, reviews changes, flags risky dependencies, and summarizes release diffs. After an alert fires, it correlates logs, builds an incident timeline, connects deployments with runtime events, retrieves runbooks, and supports root-cause work. AWS includes deployment frequency, change failure rate, and MTTR among its recommended operational measures.
Higher change volume exposes weaknesses in testing, architecture, version control, and deployment controls. A mature delivery system absorbs the throughput. A fragile one converts it into incidents.
Watch for: A falling MTTR looks excellent even when incident frequency climbs.
Read alongside: Change failure rate plus deployment frequency, MTTR plus incident frequency.
4. Where defects get caught: AI rework rate and defect escape rate
A fast team can still generate expensive work when defects travel too far down the pipeline. These two metrics show where problems get caught.
AI rework rate measures how much AI-assisted output needs substantial correction before production. Defect escape rate measures how much gets past engineering controls and reaches users.
Rework rate shows how much AI-generated output needs correction. Track rejected code, rewritten tests, corrected documentation, repair hours, and PRs sent back for major changes.
The goal is not zero rework. Watch whether AI output creates a growing review and repair burden.
A high AI acceptance rate looks impressive on a dashboard. The number tells you how often developers accept suggestions. It says nothing about what happens next. A team can accept 80% of generated suggestions and still spend a large part of the week rewriting what it accepted.
Defect escape rate adds the other axis. One team catches generated defects during review, producing measurable rework and protecting production. Another reports little rework because review is shallow, then pays for the same defects through incidents, support tickets, and emergency fixes.
GitLab surveyed 1,528 DevSecOps professionals across six countries for its report and describes the same movement: faster code creation, with the pressure landing on review, security, compliance, deployment, context, and traceability.
Watch for: Test coverage can climb without production quality improving. Track escaped-defect severity next to defect count, and review rework hours next to acceptance rates.
Read alongside: AI rework rate plus defect escape rate, with severity and remediation cost included.
A common mistake is measuring generated output before measuring the review capacity around it. A team introduces an AI coding tool, development speed rises, and six months later, the senior engineers are spending most of the week reviewing generated code. Track review time and AI rework from day one. When both climb faster than delivery performance improves, the bottleneck has moved deeper into the engineering workflow.
5. Developer time recovered
AI development productivity should answer an operational question: how much engineering capacity did the organization get back?
The unit is time, and the destination matters more than the total. A developer who saves six hours through code generation can spend those hours on architecture, security review, mentoring, technical debt, customer issues, or the debugging nobody wants.
Say an assistant saves eight hours per developer per week. Across 50 developers, the theoretical recovery is 400 hours a week. Then look at where the hours went.
The headline number changes meaning once the allocation becomes visible.
AWS recommends measuring knowledge-sharing efficiency through time spent searching for information or asking repetitive questions, alongside broader team productivity measures.
Watch for: Self-reported hours drift, and saved time disappears into new review or coordination work.
Read alongside: Developer time recovered plus post-adoption time allocation.
6. AI workflow coverage
AI adoption inside an IDE says almost nothing about engineering maturity.
A mature AI engineering workflow can extend across planning, requirements, architecture, development, testing, review, deployment, and operations. The goal is not an AI tool in every stage. The goal is identifying suitable workflows, defining controls, and connecting AI capabilities to the engineering systems already in place.
Coverage worth counting looks like this:
- AI use cases have named owners
- Teams work to defined approval rules
- AI output can be traced through the workflow
- Quality checks stay part of delivery
- Results can be measured against a baseline
- Humans keep responsibility for consequential technical decisions
GitLab found 80% of respondents saying their organizations adopted AI tools faster than the policies for governing them, and 92% reporting governance challenges with AI-generated code.
That gap is a maturity signal in itself. Tool rollout takes weeks. Ownership, integration, traceability, review rules, and measurement take considerably longer.
Watch for: Counting tools, licenses, or workflows produces an adoption KPI, not evidence of integration.
Read alongside: Delivery, quality, governance, and workflow outcomes together.
7. Cost per successful engineering outcome
AI costs do not stop at model usage. A working calculation includes tooling, infrastructure, model calls, integration, engineering review, correction, security controls, and governance work. Compare the total against a defined successful outcome.
An outcome can be a production feature, a completed migration, a resolved incident, or another clearly defined unit of value.
The metric catches an accounting mistake teams make regularly. A cheaper model does not produce cheaper software.
If Model A costs less per request and generates more review work than Model B, the lower API bill still ends at a higher cost per successful release.
Watch for: Token cost alone ignores review, rework, governance, infrastructure, and operations.
Read alongside: Cost per successful outcome plus measurable business outcomes.
8. Business value per engineering release
The engineering dashboard should connect to business results.
A faster release matters when customers use the feature. Faster incident recovery matters when availability improves. Recovered engineering time matters when teams deliver more valuable work. Useful measures include feature adoption, customer satisfaction, conversion, revenue, support volume, retention, and time from request to usable feature.
For example, releasing a feature two weeks earlier means little if adoption stays flat. If the same release reaches 20% more target users during a key sales period, the business impact is clearer.
Engineering does not own every business result. Pricing, sales, demand, and seasonality also play a role. The goal is to connect engineering performance with business outcomes.
Watch for: A business metric can move for reasons with nothing to do with engineering.
Read alongside: Business value per release plus lead time, quality, and release performance.
9. Human oversight quality
AI-generated software still needs engineering judgment. The measurement problem is evaluating human oversight without turning "a human looked at it" into another box to tick.
Useful indicators:
- Review coverage for AI-assisted changes
- The share of high-risk changes receiving qualified review
- Security findings caught before release
- The share of generated changes with traceable ownership.
A review process worth measuring answers who reviewed the change, what risk it carried, which automated checks ran, what evidence supported approval, and whether the change can be traced after release. These controls matter more as AI moves past code completion into agents handling larger portions of the engineering workflow.
Watch for: 100% review coverage still means weak oversight when reviewers lack context or time.
Read alongside: Oversight coverage plus escaped defects, AI rework, and change failure rate.
10. Engineering process maturity
The last metric is less a number than a question: can the organization sustain AI-driven software development?
DORA's research supplies the principle. AI amplifies existing engineering strengths and weaknesses. Strong internal platforms, small-batch work, safety nets, user focus, and healthy organizational practices give teams the conditions to convert adoption into delivery performance.
A mature team knows where AI performs well, where engineers take over, which controls apply at each risk level, and how to measure results without rewarding superficial output.
The progression runs: tools in developers' hands, AI integrated into selected tasks with defined controls, AI connected to repositories, CI/CD, observability, and testing systems, and finally delivery speed, quality, capacity, cost, and business value measured together. Only the last level tells leadership anything about return.
Watch for: Licenses, active users, and tool counts make adoption look mature before the process has changed.
Read alongside: Workflow coverage plus delivery, quality, cost, and business metrics.
How to read the scorecard without fooling yourself
Ten metrics introduce a new risk: teams optimize the dashboard and lose the software. Look at the relationships between numbers.
A single KPI gives a clean answer to the wrong question. A deployment increase reads better when failure rates hold. Time recovery reads better when lead time falls with it. Lower rework reads better when escaped-defect severity stays flat.
Build the baseline before you set the AI target
The TYMIQ team adheres to the following principle: AI measurement works when teams have a pre-AI baseline.
Capture the baseline for lead time, deployment frequency, PR cycle time, change failure rate, MTTR, defect escape rate, review effort, developer time allocation, AI-related engineering cost, and the business outcomes attached to them. Then compare similar periods, teams, or workflows, which makes it easier to separate improvement from normal variation.
A useful target reads like this:
"Reduce median lead time for this workflow by 20% without increasing change failure rate."
A weak one reads like this:
"Increase AI-generated code by 30%."
The first describes an engineering outcome. The second describes activity.
The CTO scorecard at a glance – just copy and apply in your workflow:
The scorecard gives leadership one page showing delivery performance, quality, engineering capacity, process maturity, and business value at the same time.
Measure the software, not the AI
A useful AI engineering scorecard can fit on one dashboard. The hard part comes before the dashboard: choosing outcome-based metrics and setting a baseline before AI changes the workflow.
The strongest KPIs connect three layers: AI activity, engineering performance, and business results. Tool adoption and generated output should lead to better delivery, quality, and capacity, which should translate into measurable business outcomes.
Adoption counts and prompt volume still have a place in operational reporting. They matter when read alongside the numbers showing whether AI is helping the business deliver better software.
.webp)
.svg)
