An agent system's scale and its usefulness are related questions. A larger team may explore more possibilities or finish independent tasks sooner. To judge whether that capacity helps, follow the work through acceptance, release and recovery.
This guide is a scorecard for your own operation. For documented measurements from other organizations, including concurrent workers, cumulative runs and adoption, use the agent scale reference.
Which five software factory metrics should a PM track?
1. Time to a useful result
Measure from the moment a task brief is accepted to the moment the checked result reaches its intended user or system.
verified in production at − work accepted at
Split the total into waiting, agent work, human review, release wait, and recovery. This shows whether another agent shortens delivery or only creates a longer review queue.
2. Results accepted without material rework
accepted without a major correction / completed results
Define “material” before the pilot. A spelling fix may not count. Reworking the behavior, product decision, or architecture should. This is more informative than counting every generated patch.
3. Problems found after approval
Track regressions, reverts, incidents, security findings, and customer-visible defects that escape the proof loop. Pair the rate with severity and recovery time. Faster output is not an improvement when it creates expensive cleanup.
4. Human attention per accepted result
Add the time people spend clarifying the brief, steering the run, reviewing the change, resolving conflicts, and recovering from failure. Human attention is often the scarce resource. An agent that saves compute but consumes more senior review time can have negative value.
5. Total cost per accepted result
(models + compute + tools + review + recovery) / accepted results
Use a product-owned unit such as a released bug fix, resolved alert, verified migration, or merged dependency update. Token cost belongs in the numerator, but it is not the whole cost. HarnessTax shows harness choice moving token cost more than success on two public coding-agent benches. Use that as a prompt to compare wrappers, not as a substitute for measuring accepted results.
How do these metrics connect to ROI?
The five numbers describe delivery performance. They do not prove the team chose the right work. Pair them with the product result the change was meant to improve, such as task success, support contacts, conversion, retention, or reliability.
The 2025 DORA research studies AI use alongside throughput, stability, platform quality, user focus, and organizational conditions. Its findings are associations in the studied population, not a promise that adding agents will cause a particular business result. The practical lesson is to measure speed and stability together.
What are AI agent evals?
An eval is a realistic, repeatable scenario with a scoring rule. A set of related evals is an eval suite.
For a checkout agent, one scenario might say: “When a discount code expires, keep the cart and explain the next step.” A test can check the cart. A screenshot can show the message. A person or a model grader whose scores have been checked against human judgment can assess clarity. The scenario, inputs, result, and scoring decision stay together.
Teams often separate three kinds of eval:
- Regression evals protect behavior that already works.
- Launch evals test whether a new capability is ready for its intended users.
- Headroom evals contain harder cases the system cannot solve yet, so progress remains measurable.
Notion's Token Town interview describes this vocabulary and its use of self-verification. That is one operator's account, not a standard.
How should scale and autonomy be measured?
Record concurrent workers, total runs, retries, elapsed time and compute separately. If you increase the worker count, compare the accepted output and total cost on the same kind of task. Watch for duplicate work, conflicts and a growing review queue. A peak worker count describes capacity at a moment; it does not establish sustained throughput.
Adoption needs its own denominator. Record the share of eligible teams using the system, or the share of eligible work completed through it, with the measurement window. Access to an assistant and recurring use of an automated workflow are different observations.
State the task class and success rate. “The agent works for six hours” says little if half the runs are discarded. METR's time-horizon research models capability as a probability across tasks that take people different amounts of time. Its authors also warn that benchmark movement does not translate cleanly into real-world usefulness.
Record task-complete rate, retries, interventions, rejected runs, cost, review time, and failures found after release. Add token yield, duplicate work, or integration repair only when the core scorecard exposes a problem worth diagnosing.
Uber's August 2026 operating account reports its own agent adoption, skills, executions, costs, managed-agent outcomes, and quality signals. Those figures belong to Uber. The reusable idea is the measurement design: connect spend to an accepted unit of work and keep a quality signal beside it.
A weekly review a PM can run
- Pick three to five accepted results and inspect their briefs, run records, proof, review, and release.
- Compare time, rework, escaped defects, human attention, and total cost with the prior process.
- Find the largest wait or repeated failure.
- Improve one reusable part of the system: the task brief, supplied context, agent harness, proof loop, or guardrail.
- Run the same kind of work again and check whether the improvement transfers.
Keep the task mix and acceptance rules visible as the system grows. Otherwise a change in the workload can look like a change in the system's performance.