--- schema: "swft.publication/v1" id: "software-factory-metrics" title: "Software factory metrics that matter" description: "Measure AI software factory ROI with delivery time, rework, escaped defects, human review time, cost per accepted change, and agent evals." summary: "Measure an AI software factory by useful results, not code or token volume. Track time to a released result, material rework, defects found after approval, human attention, and total cost per accepted change. Pair all five with the product outcome the change was meant to improve." canonical: "https://swft.io/software-factory-metrics" author: "SWFT Editorial" author_type: "Organization" published: "2026-08-31" modified: "2026-09-02" kind: "guide" section: "Guides" tags: ["software factory metrics", "AI coding agent ROI", "coding agent productivity", "AI agent evals", "DORA metrics"] evidence_labels: ["INFERENCE", "SELF-REPORT", "STUDY"] source_ids: ["dora-2025", "dora-2025-errata", "latent-space-notion-token-town", "metr-time-horizons", "uber-efficient-software-factory"] authorship_disclosure: "AI-drafted from the cited public sources and independently checked by a second AI editorial-review agent (Codex) for source fit, claim boundaries, overlap, and reader utility. SWFT Editorial is responsible for corrections." --- # Software factory metrics that matter Five numbers for judging speed, rework, quality, human attention, and the real cost of AI-assisted delivery. > **Authorship:** AI-drafted from the cited public sources and independently checked by a second AI editorial-review agent (Codex) for source fit, claim boundaries, overlap, and reader utility. SWFT Editorial is responsible for corrections. ## Quick answer Measure an AI software factory by useful results, not code or token volume. Track time to a released result, material rework, defects found after approval, human attention, and total cost per accepted change. Pair all five with the product outcome the change was meant to improve. Imagine an agent opens ten pull requests, reviewers reject six, and two accepted changes later cause incidents. The activity chart looks busy, but the product team gained little. Useful measurement starts with the results that reach users and survive contact with production. ## Which five software factory metrics should a PM track? ### 1. Time to a useful result Measure from the moment a task brief is accepted to the moment the checked result reaches its intended user or system. `verified in production at − work accepted at` Split the total into waiting, agent work, human review, release wait, and recovery. This shows whether another agent shortens delivery or only creates a longer review queue. ### 2. Results accepted without material rework `accepted without a major correction / completed results` Define “material” before the pilot. A spelling fix may not count. Reworking the behavior, product decision, or architecture should. This is more informative than counting every generated patch. ### 3. Problems found after approval Track regressions, reverts, incidents, security findings, and customer-visible defects that escape the proof loop. Pair the rate with severity and recovery time. Faster output is not an improvement when it creates expensive cleanup. ### 4. Human attention per accepted result Add the time people spend clarifying the brief, steering the run, reviewing the change, resolving conflicts, and recovering from failure. Human attention is often the scarce resource. An agent that saves compute but consumes more senior review time can have negative value. ### 5. Total cost per accepted result `(models + compute + tools + review + recovery) / accepted results` Use a product-owned unit such as a released bug fix, resolved alert, verified migration, or merged dependency update. Token cost belongs in the numerator, but it is not the whole cost. ## How do these metrics connect to ROI? The five numbers describe delivery performance. They do not prove the team chose the right work. Pair them with the product result the change was meant to improve, such as task success, support contacts, conversion, retention, or reliability. The [2025 DORA research](https://dora.dev/research/2025/dora-report/) studies AI use alongside throughput, stability, platform quality, user focus, and organizational conditions. Its findings are associations in the studied population, not a promise that adding agents will cause a particular business result. The practical lesson is to measure speed and stability together. ## What are AI agent evals? An **eval** is a realistic, repeatable scenario with a scoring rule. A set of related evals is an **eval suite**. For a checkout agent, one scenario might say: “When a discount code expires, keep the cart and explain the next step.” A test can check the cart. A screenshot can show the message. A person or a model grader whose scores have been checked against human judgment can assess clarity. The scenario, inputs, result, and scoring decision stay together. Teams often separate three kinds of eval: - **Regression evals** protect behavior that already works. - **Launch evals** test whether a new capability is ready for its intended users. - **Headroom evals** contain harder cases the system cannot solve yet, so progress remains measurable. [Notion's Token Town interview](https://www.latent.space/p/notion) describes this vocabulary and its use of self-verification. That is one operator's account, not a standard. ## How should autonomy be measured? State the task class and success rate. “The agent works for six hours” says little if half the runs are discarded. [METR's time-horizon research](https://metr.org/blog/2025-03-19-measuring-ai-ability-to-complete-long-tasks/) models capability as a probability across tasks that take people different amounts of time. Its authors also warn that benchmark movement does not translate cleanly into real-world usefulness. Record task-complete rate, retries, interventions, rejected runs, cost, review time, and failures found after release. Add token yield, duplicate work, or integration repair only when the core scorecard exposes a problem worth diagnosing. Uber's [August 2026 operating account](https://www.uber.com/us/en/blog/efficient-software-factory/) reports its own agent adoption, skills, executions, costs, managed-agent outcomes, and quality signals. Those figures belong to Uber. The reusable idea is the measurement design: connect spend to an accepted unit of work and keep a quality signal beside it. ## A weekly review a PM can run 1. Pick three to five accepted results and inspect their briefs, run records, proof, review, and release. 2. Compare time, rework, escaped defects, human attention, and total cost with the prior process. 3. Find the largest wait or repeated failure. 4. Improve one reusable part of the system: the task brief, supplied context, agent harness, proof loop, or guardrail. 5. Run the same kind of work again and check whether the improvement transfers. The goal is a delivery system that produces more useful results without hiding rework, risk, or human attention. ## How we know - **Independent study (STUDY)** DORA reports AI as an amplifier and analyzes throughput, stability, platform, user-focus, and organizational conditions together. Sources: [State of AI-assisted Software Development 2025](https://dora.dev/research/2025/dora-report/); [DORA Research: 2025 Errata](https://dora.dev/research/2025/errata/). - **Independent study (STUDY)** METR measures autonomous capability as success probability over tasks characterized by human completion time. Sources: [Measuring AI Ability to Complete Long Tasks](https://metr.org/blog/2025-03-19-measuring-ai-ability-to-complete-long-tasks/). - **First-party report (SELF-REPORT)** Uber reports its own adoption, skills, daily executions, unit economics, managed-agent outcomes, and quality signals. Sources: [Running a Software Factory Efficiently at Uber Scale](https://www.uber.com/us/en/blog/efficient-software-factory/). - **Analysis (INFERENCE)** SWFT's five-number scorecard joins delivery speed, material rework, escaped defects, human attention, and total cost, then connects them to a product result. Sources: [State of AI-assisted Software Development 2025](https://dora.dev/research/2025/dora-report/); [Measuring AI Ability to Complete Long Tasks](https://metr.org/blog/2025-03-19-measuring-ai-ability-to-complete-long-tasks/); [Running a Software Factory Efficiently at Uber Scale](https://www.uber.com/us/en/blog/efficient-software-factory/); [Notion's Token Town](https://www.latent.space/p/notion). ## Sources - **Independent study (STUDY)** [State of AI-assisted Software Development 2025](https://dora.dev/research/2025/dora-report/) — DORA; accessed 2026-08-31. DORA's research report on AI adoption, throughput, stability, platform quality, and organizational conditions. - **Observed artifact (OBS)** [DORA Research: 2025 Errata](https://dora.dev/research/2025/errata/) — DORA; updated 2025-11-24; accessed 2026-08-31. The official correction record for the 2025 report and its current versioning practice. - **First-party report (SELF-REPORT)** [Notion's Token Town](https://www.latent.space/p/notion) — Latent Space; accessed 2026-09-01. An operator interview about regression, launch, and headroom evals, self-verification, agent skills, and the changing boundary between product and engineering work. - **Independent study (STUDY)** [Measuring AI Ability to Complete Long Tasks](https://metr.org/blog/2025-03-19-measuring-ai-ability-to-complete-long-tasks/) — METR; published 2025-03-19; accessed 2026-08-31. Study framing agent capability as a reliability curve over task duration rather than a binary autonomy claim. - **First-party report (SELF-REPORT)** [Running a Software Factory Efficiently at Uber Scale](https://www.uber.com/us/en/blog/efficient-software-factory/) — Uber Engineering; published 2026-08-27; accessed 2026-08-31. Uber's account of factory layers, adoption, token economics, managed agents, quality signals, and context infrastructure. ## Read next - [How an AI software factory works](/software-factory-architecture) - [How to build a software factory](/how-to-build-a-software-factory) - [Inside Uber's AI software factory: measuring cost and quality at scale](/companies/uber-software-factory)