---
schema: "swft.publication/v1"
id: "build-software-factory"
title: "How to build a software factory"
description: "A PM-friendly guide to piloting an AI software factory with a task brief, agent harness, sandbox, evals, approvals, metrics, and recovery."
summary: "Pilot an AI software factory with one recurring, low-risk job. Define what done means, measure today's process, prepare a safe agent harness, build realistic evals, name the decisions that need a person, and expand autonomy only after the evidence improves."
canonical: "https://swft.io/how-to-build-a-software-factory"
author: "SWFT Editorial"
author_type: "Organization"
published: "2026-08-31"
modified: "2026-09-02"
kind: "guide"
section: "Guides"
tags: ["how to build a software factory", "AI software factory pilot", "agent harness", "AI agent evals", "coding agents"]
evidence_labels: ["INFERENCE", "SELF-REPORT", "STUDY"]
source_ids: ["ai-engineer-harness-failure", "anthropic-agent-evals", "anthropic-effective-agents", "dora-2025", "metr-time-horizons"]
authorship_disclosure: "AI-drafted from the cited public sources and independently checked by a second AI editorial-review agent (Codex) for source fit, claim boundaries, overlap, and reader utility. SWFT Editorial is responsible for corrections."
---

# How to build a software factory

A six-stage pilot for one recurring job, with a clear task brief, safe agent harness, realistic evals, and named human decisions.

> **Authorship:** AI-drafted from the cited public sources and independently checked by a second AI editorial-review agent (Codex) for source fit, claim boundaries, overlap, and reader utility. SWFT Editorial is responsible for corrections.

## Quick answer

Pilot an AI software factory with one recurring, low-risk job. Define what done means, measure today's process, prepare a safe agent harness, build realistic evals, name the decisions that need a person, and expand autonomy only after the evidence improves.

The first pilot should teach the team whether one production line can repeat safely. An expired-discount-code bug, a dependency update, or a familiar alert is easier to learn from than a broad feature with unsettled product decisions.

## 1. Choose one repeatable job

Choose work that already matters, is narrow enough to check, common enough to repeat, and safe enough to recover when the system makes a mistake. Good candidates include documentation corrections, narrow bug fixes, and test-backed migrations.

Write the outcome in product language. “Fix expired discount codes” becomes “Keep the cart, explain why the code failed, never charge unexpectedly, and preserve existing checkout behavior.”

## 2. Create the task brief

The **task brief** is the specification and definition of done. It should name:

- the user or system outcome;
- what the agent may and may not change;
- known product decisions and open questions;
- risk, owner, and required approvals;
- checks that prove the result;
- how to release and recover.

If the agent would have to invent a pricing, legal, security, or experience decision, the brief is not ready.

## 3. Record today's baseline

Run several representative examples through the current process. Measure delivery time, human effort, rework, failures, recovery, and total cost. Keep the examples. A pilot needs a comparison with real work, not a polished demo.

## 4. Prepare the agent harness

The **agent harness** should provide the right repository state, tools, supplied context, and a sandbox with limited permissions. Pin important tool versions. Control credentials and network access. Keep a run record of what the agent saw, changed, called, and checked.

Use ordinary software for exact steps. Use a fixed AI workflow when the path is known but interpretation helps. Use an adaptive coding agent when the path is uncertain and the outcome can still be verified. [Anthropic's practitioner guide](https://www.anthropic.com/engineering/building-effective-agents) recommends adding agentic complexity only when it improves the result.

## 5. Build the proof loop and human approval gates

Create realistic eval cases before expanding autonomy. Include common examples, important edge cases, known past failures, and at least one case that should make the agent stop.

Combine deterministic tests with human judgment where product taste or ambiguity matters. Preserve an **evidence package** containing the brief, relevant context, run record, exact artifact, checks, approvals, release, and production result.

Name every **human approval gate**, which SWFT calls an **attention gate**. A person might approve a product choice, sensitive data access, security exception, merge, or production release. Give the run a time and cost budget, explicit write boundaries, stop conditions, and a recoverable path for external actions.

## 6. Run the pilot and improve one boundary at a time

Review accepted and rejected runs together. Classify repeated problems by the part of the system that should improve:

- unclear brief;
- missing or stale context;
- model limitation;
- poor tool;
- unreliable workspace;
- weak eval or test;
- permission conflict;
- release failure;
- wrong product assumption.

Fix the reusable boundary and add the failure to the eval suite. Correcting one patch is recovery. Preventing the same class of failure is learning.

Expand the task scope, permissions, parallel agents, or proactive triggers one dimension at a time. [METR's time-horizon research](https://metr.org/blog/2025-03-19-measuring-ai-ability-to-complete-long-tasks/) is a reminder that autonomy is probabilistic. State the kind of work and reliability you measured.

## Sample pilot brief

| Field | Example |
| --- | --- |
| Goal | Explain expired discount codes without losing the cart |
| Allowed scope | Checkout error handling, copy, and tests |
| Must not change | Pricing, payment capture, or discount eligibility |
| Proof | Regression tests, checkout scenario, screenshot, human copy review |
| Human approval gates | Any pricing ambiguity, security finding, merge, production release |
| Recovery | Revert the exact released change and preserve the run record |

## Minimum launch checklist

- [ ] One recurring job with an owner
- [ ] A task brief with a clear definition of done
- [ ] Real baseline examples and measurements
- [ ] A safe, reproducible agent harness
- [ ] Declared context and permissions
- [ ] Realistic evals and a complete evidence package
- [ ] Named human approval gates, budgets, and stop rules
- [ ] A checked release and recovery path
- [ ] A weekly review of results, quality, attention, cost, and learning

The first production line is successful when the next valid request follows the same understandable path and a failure leaves enough evidence to improve that path.

## Questions product teams ask

### How long should an AI software factory pilot run?

Run enough representative cases to compare delivery time, rework, quality, human attention, and total cost with the existing process. A fixed calendar length matters less than covering normal work, edge cases, failures, and recovery.

## How we know

- **First-party report (SELF-REPORT)** Anthropic recommends simple composable patterns, ground truth from environments, and explicit evaluation for agent systems. Sources: [Building effective agents](https://www.anthropic.com/engineering/building-effective-agents); [Demystifying evals for AI agents](https://www.anthropic.com/engineering/demystifying-evals-for-ai-agents).
- **Independent study (STUDY)** METR models autonomous task completion probabilistically across task durations. Sources: [Measuring AI Ability to Complete Long Tasks](https://metr.org/blog/2025-03-19-measuring-ai-ability-to-complete-long-tasks/).
- **Independent study (STUDY)** DORA finds the surrounding platform and organizational system material to AI-assisted delivery outcomes. Sources: [State of AI-assisted Software Development 2025](https://dora.dev/research/2025/dora-report/).
- **Analysis (INFERENCE)** The six-stage pilot, task brief, evidence package, and promotion rules are SWFT's operating synthesis for a minimum useful factory. Sources: [Building effective agents](https://www.anthropic.com/engineering/building-effective-agents); [Demystifying evals for AI agents](https://www.anthropic.com/engineering/demystifying-evals-for-ai-agents); [State of AI-assisted Software Development 2025](https://dora.dev/research/2025/dora-report/); [Measuring AI Ability to Complete Long Tasks](https://metr.org/blog/2025-03-19-measuring-ai-ability-to-complete-long-tasks/); [Harness Engineering Is Not Enough: Why Software Factories Fail](https://ai.engineer/talks/Ib5GBkD555M-harness-engineering-is-not-enough-why-software).

## Sources

- **First-party report (SELF-REPORT)** [Harness Engineering Is Not Enough: Why Software Factories Fail](https://ai.engineer/talks/Ib5GBkD555M-harness-engineering-is-not-enough-why-software) — AI Engineer; published 2026-07-23; accessed 2026-09-01. A practitioner counterpoint arguing that faster generation still needs architecture, review, maintainability, and human judgment.
- **First-party report (SELF-REPORT)** [Demystifying evals for AI agents](https://www.anthropic.com/engineering/demystifying-evals-for-ai-agents) — Anthropic; published 2026-01-09; accessed 2026-08-31. Practitioner guidance on task suites, graders, transcripts, and evaluation lifecycle for agents.
- **First-party report (SELF-REPORT)** [Building effective agents](https://www.anthropic.com/engineering/building-effective-agents) — Anthropic; published 2024-12-19; accessed 2026-08-31. Practitioner guidance distinguishing fixed workflows from agents and describing common orchestration patterns; Anthropic now warns that the 2024 tooling landscape described there has changed.
- **Independent study (STUDY)** [State of AI-assisted Software Development 2025](https://dora.dev/research/2025/dora-report/) — DORA; accessed 2026-08-31. DORA's research report on AI adoption, throughput, stability, platform quality, and organizational conditions.
- **Independent study (STUDY)** [Measuring AI Ability to Complete Long Tasks](https://metr.org/blog/2025-03-19-measuring-ai-ability-to-complete-long-tasks/) — METR; published 2025-03-19; accessed 2026-08-31. Study framing agent capability as a reliability curve over task duration rather than a binary autonomy claim.

## Read next

- [How an AI software factory works](/software-factory-architecture)
- [Software factory metrics that matter](/software-factory-metrics)
- [Methods and corrections](/methods)