Imagine an agent opens ten pull requests, reviewers reject six, and two accepted changes later cause incidents. The activity chart looks busy, but the product team gained little. Useful measurement starts with the results that reach users and survive contact with production.
Which five software factory metrics should a PM track?
1. Time to a useful result
Measure from the moment a task brief is accepted to the moment the checked result reaches its intended user or system.
verified in production at − work accepted at
Split the total into waiting, agent work, human review, release wait, and recovery. This shows whether another agent shortens delivery or only creates a longer review queue.
2. Results accepted without material rework
accepted without a major correction / completed results
Define “material” before the pilot. A spelling fix may not count. Reworking the behavior, product decision, or architecture should. This is more informative than counting every generated patch.
3. Problems found after approval
Track regressions, reverts, incidents, security findings, and customer-visible defects that escape the proof loop. Pair the rate with severity and recovery time. Faster output is not an improvement when it creates expensive cleanup.
4. Human attention per accepted result
Add the time people spend clarifying the brief, steering the run, reviewing the change, resolving conflicts, and recovering from failure. Human attention is often the scarce resource. An agent that saves compute but consumes more senior review time can have negative value.
5. Total cost per accepted result
(models + compute + tools + review + recovery) / accepted results
Use a product-owned unit such as a released bug fix, resolved alert, verified migration, or merged dependency update. Token cost belongs in the numerator, but it is not the whole cost.
How do these metrics connect to ROI?
The five numbers describe delivery performance. They do not prove the team chose the right work. Pair them with the product result the change was meant to improve, such as task success, support contacts, conversion, retention, or reliability.
The 2025 DORA research studies AI use alongside throughput, stability, platform quality, user focus, and organizational conditions. Its findings are associations in the studied population, not a promise that adding agents will cause a particular business result. The practical lesson is to measure speed and stability together.
What are AI agent evals?
An eval is a realistic, repeatable scenario with a scoring rule. A set of related evals is an eval suite.
For a checkout agent, one scenario might say: “When a discount code expires, keep the cart and explain the next step.” A test can check the cart. A screenshot can show the message. A person or a model grader whose scores have been checked against human judgment can assess clarity. The scenario, inputs, result, and scoring decision stay together.
Teams often separate three kinds of eval:
- Regression evals protect behavior that already works.
- Launch evals test whether a new capability is ready for its intended users.
- Headroom evals contain harder cases the system cannot solve yet, so progress remains measurable.
Notion's Token Town interview describes this vocabulary and its use of self-verification. That is one operator's account, not a standard.
How should autonomy be measured?
State the task class and success rate. “The agent works for six hours” says little if half the runs are discarded. METR's time-horizon research models capability as a probability across tasks that take people different amounts of time. Its authors also warn that benchmark movement does not translate cleanly into real-world usefulness.
Record task-complete rate, retries, interventions, rejected runs, cost, review time, and failures found after release. Add token yield, duplicate work, or integration repair only when the core scorecard exposes a problem worth diagnosing.
Uber's August 2026 operating account reports its own agent adoption, skills, executions, costs, managed-agent outcomes, and quality signals. Those figures belong to Uber. The reusable idea is the measurement design: connect spend to an accepted unit of work and keep a quality signal beside it.
A weekly review a PM can run
- Pick three to five accepted results and inspect their briefs, run records, proof, review, and release.
- Compare time, rework, escaped defects, human attention, and total cost with the prior process.
- Find the largest wait or repeated failure.
- Improve one reusable part of the system: the task brief, supplied context, agent harness, proof loop, or guardrail.
- Run the same kind of work again and check whether the improvement transfers.
The goal is a delivery system that produces more useful results without hiding rework, risk, or human attention.