Erik Caldwell
Connect on LinkedIn
Menu

The Topic Catalog · 20

AI Measurement & Business Value

Last updated · 10 min read

20.1 Engineering Metrics in the AI Era (DORA Metrics + AI-Specific Measures)

Priority: Must Understand

Executive Definition: DORA's four/five keys: deployment frequency, lead time for changes, change failure rate, time to restore service, and (in more recent research) measures of rework/churn: remain the standard for software delivery performance, but DORA's own current research is explicit that AI coding assistants are changing what these metrics mean: code gets generated faster, but verification, review, and integration overhead grow, so throughput metrics alone can improve while real delivery quality does not. DORA's research specifically warns against "tokenmaxxing" (treating AI token/prompt volume as a performance indicator) and recommends selecting from DORA, SPACE, or DevEx frameworks based on what your organization actually needs to learn, not defaulting to whichever is easiest to instrument.

Why It Matters: Engineering leaders under pressure to show "AI ROI" are the group most likely to reach for the easiest available number (lines of code generated, percentage of code "AI-assisted," tool licenses deployed) and DORA's research is a direct, credible warning that these are exactly the wrong signals, because AI can inflate them while quietly increasing verification burden, skill degradation, and change failure rate elsewhere in the pipeline.

What I Need to Understand:

  • DORA's core metrics (deployment frequency, lead time, change failure rate, time to restore) still matter, but interpret them alongside stability metrics specifically: AI acceleration of the "throughput" side without matching improvement in "stability" metrics is a warning sign, not a win.
  • DORA research documents a real paradox: developers using AI tools report higher satisfaction while also reporting they spend less time on work they consider valuable: satisfaction and productivity are not the same signal, and a rise in one doesn't validate the other.
  • "Tokenmaxxing" (rewarding or measuring AI adoption by volume of tokens/prompts consumed) is explicitly flagged by DORA as a dangerous trap, because it incentivizes usage rather than outcomes (this is the engineering-specific instance of the broader Topic 12 problem).
  • AI coding assistants introduce hidden costs not captured by classic delivery metrics: verification overhead, skill degradation risk for less experienced engineers, and integration friction: these need their own tracking, not just an assumption that faster code generation equals faster delivery.
  • No single framework (DORA vs. SPACE vs. DevEx) is sufficient alone; DORA's own guidance is to choose the framework(s) that match what you're actually trying to learn about your organization, and to expect that to include qualitative developer-experience signals alongside quantitative delivery metrics.

Questions I Should Be Able to Ask My Team:

  1. Since introducing AI coding tools, have our change failure rate and time to restore moved in the same direction as our deployment frequency and lead time, or is throughput up while stability is flat or worse?
  2. Are we measuring or incentivizing anything by raw AI usage volume (tokens, prompts, "percent AI-assisted code"), and if so, can we defend that against DORA's tokenmaxxing warning?
  3. Do we have any signal on verification overhead or code churn/rework tied to AI-generated code, or are we only looking at generation speed?

Technologies / Standards / Companies to Know: DORA (DevOps Research and Assessment), SPACE framework, DevEx framework, AI coding assistants (GitHub Copilot, Claude Code, Cursor, etc.: evaluate neutrally per vendor)

Recommended Learning:

  • DORA Insights: the primary source for current DORA research and reports.
  • DORA research on AI: DORA's dedicated body of work on AI's effect on software delivery, including the ROI of AI-assisted development report.

Time Investment: 2-3 hours

20.2 Knowledge-Worker Productivity Metrics

Priority: Must Understand

Executive Definition: Measuring AI's effect on non-engineering knowledge work requires task-level output metrics (units completed, quality, time-to-resolution) tied to a credible baseline and, critically, segmented by worker skill/experience level: because the best available field research shows AI's productivity effect is highly uneven across a workforce, not a uniform lift you can apply as a flat multiplier.

Why It Matters: The most rigorous field study to date (a large-scale study of customer support agents using a generative AI assistant) found a 14% average productivity gain but with the gain concentrated almost entirely among novice and lower-skilled workers (~34% gain) while experienced workers saw little to no measurable benefit: an average number here actively hides the story, and any executive reporting a single average productivity figure across a whole knowledge-worker population is very likely obscuring who is actually benefiting and why.

What I Need to Understand:

  • Aggregate/average productivity figures across a knowledge-worker population are close to meaningless without segmentation by experience or skill level: the underlying research shows the effect is concentrated, not uniform.
  • The strongest documented mechanism for AI productivity gains in knowledge work is diffusion of best practices from top performers to less experienced workers, effectively compressing the learning curve: this is a different and more measurable claim than generic "AI makes people more productive."
  • Task-level output metrics (issues resolved per hour, time-to-resolution, quality/error rate, customer satisfaction) are more defensible than time-saved self-reports, which are notoriously unreliable and inflated.
  • Retention and skill-development effects (does AI use correlate with faster ramp-up for new hires, lower attrition) are real, measurable secondary benefits documented in the research and worth tracking alongside raw output.
  • Any productivity claim needs a comparison group or a credible pre/post baseline: anecdote-driven claims ("teams say they're faster") are not evidence and should be treated skeptically at the executive level.

Questions I Should Be Able to Ask My Team:

  1. When we report a knowledge-worker productivity gain from AI, is it an average across everyone, or do we know which segments (tenure, skill level) are actually driving it?
  2. Do we have task-level output metrics with a real baseline for any AI-assisted knowledge work function, or are we relying on self-reported time savings?
  3. Are we seeing evidence that AI use is compressing ramp-up time for new hires or narrowing the gap between our best and average performers: the effect the strongest research predicts?

Technologies / Standards / Companies to Know: Task-level output/quality metrics, controlled rollout / A-B methodology for internal AI tools, National Bureau of Economic Research (NBER) field-experiment methodology

Recommended Learning:

Time Investment: 1 hour

20.3 Enterprise Value Metrics (Revenue Enabled, Cost Reduction, Capacity Released, Adoption/Workflow Penetration)

Priority: Must Understand

Executive Definition: Enterprise AI value should be tracked through four outcome-linked categories: revenue enabled (new revenue or deals directly attributable to AI-enabled capability), cost reduction (measured, realized cost removed from a process, not projected savings), capacity released (hours or headcount-equivalent freed up and (critically) what was actually done with that freed capacity), and adoption/workflow penetration (the percentage of a given workflow's volume actually running through the AI path, not the percentage of people with a license). These are outcome metrics; everything in Topic 12 is what they are not.

Why It Matters: Most enterprise AI reporting collapses into activity metrics (usage, engagement, satisfaction scores) because they're easy to instrument, but none of them answer the question a board or CFO actually asks: did this change the economics of the business. Capacity released is the category most often mismeasured or left untracked entirely: freeing up hours that are never redeployed to higher-value work is not value capture, it's a cost center running under capacity, and reporting the hours freed without reporting what happened to them is close to fabrication.

What I Need to Understand:

  • Revenue enabled needs a defensible attribution chain (which deals, which capability, what would have happened otherwise): "AI-related revenue" claims without that chain are marketing, not measurement.
  • Cost reduction should be realized (removed from the budget) not merely modeled/projected: a business case showing hypothetical savings is not the same claim as a finance-verified reduction.
  • Capacity released is only a real value metric when paired with a redeployment answer: what is now being done with the freed time/headcount: absorbed into other work, redeployed to new initiatives, or actually reduced from the cost base. Track both halves or the metric is incomplete.
  • Adoption/workflow penetration should be measured as share of eligible task/transaction volume actually flowing through the AI-enabled path, not share of licensed seats or logged-in users (see Topic 12): penetration of the workflow, not penetration of the roster.
  • These four categories should roll up to a small, consistent executive dashboard reviewed on a cadence (quarterly is reasonable) rather than a sprawling set of team-specific metrics that can't be compared or aggregated.

Questions I Should Be Able to Ask My Team:

  1. For every capacity-released claim we report, can we show what the freed time/headcount was actually redeployed to, or are we just reporting hours saved with no destination?
  2. What percentage of the eligible transaction volume for this workflow is actually running through the AI-enabled path, as opposed to what percentage of licensed users have logged in?
  3. Walk me through the attribution chain for our largest AI-attributed revenue or cost-reduction claim: what would we expect to see if it were true, and have we checked?

Technologies / Standards / Companies to Know: Workflow-level instrumentation/analytics, finance-verified savings tracking (vs. modeled), DORA's "ROI of AI-Assisted Software Development" methodology as a template for rigor

Recommended Learning:

Time Investment: 2-3 hours

20.4 Why Prompt Counts & Seat/User Counts Are Vanity Metrics

Priority: Should Understand

Executive Definition: Prompt volume, login counts, and licensed-seat totals measure whether people touched a tool, not whether anything about the business changed as a result. They are the AI-era equivalent of "page views": trivially gameable, easy to report, and structurally incapable of answering whether a workflow got faster, cheaper, better, or eliminated. Any AI reporting that leads with these numbers should be treated as unverified until translated into the outcome metrics in Topic 11.

Why It Matters: These are the numbers every AI vendor and most internal teams default to reporting because they require no attribution work and always trend upward: which is precisely why they're worthless as evidence of value and dangerous as a basis for investment decisions.

What I Need to Understand:

  • A seat count measures procurement, not transformation. A prompt count measures curiosity, not capability change. Neither tells you whether a single workflow got faster, cheaper, or better: that is the sentence to hold every AI status report against.
  • DORA's own research names the engineering-specific version of this problem directly: "tokenmaxxing" (treating AI token/prompt spend as a performance indicator) as a dangerous trap that rewards volume over outcome (see Topic 9); the same logic applies to any usage count, in any function.
  • Usage metrics are trivially inflatable without any underlying behavior change: mandate logins, auto-generate prompts, count every keystroke as "engagement": none of it requires the workflow to actually improve.
  • The NBER field study on AI productivity (Topic 10) shows access to a tool and realized productivity gain are entirely different things that vary enormously by user: a company reporting "80% of employees are active AI users" has told you nothing about whether those users are the 34%-gain novices or the near-zero-gain experts, or anything in between.
  • The correct question is never "how much was it used": it's "which workflow changed, by how much, and what's the evidence." If a metric can go up while zero business outcomes change, it is a vanity metric, full stop.

Questions I Should Be Able to Ask My Team:

  1. When someone reports "X% of employees are active AI users" or "Y prompts sent last month," what workflow-level outcome are they actually claiming, and can they show it?
  2. If we doubled our prompt volume or seat count next quarter with zero change to any workflow's speed, cost, or quality, would our current reporting even notice? If not, our reporting is broken.
  3. For every usage metric in our current AI dashboard, is there a corresponding outcome metric (from Topic 11) it's supposed to be a proxy for, and does the proxy actually correlate with the outcome in our own data?

Technologies / Standards / Companies to Know: DORA's "tokenmaxxing" critique, outcome-based analytics vs. engagement analytics, workflow-level instrumentation

Recommended Learning:

Time Investment: 30 minutes