The first pilot should teach the team whether one production line can repeat safely. An expired-discount-code bug, a dependency update, or a familiar alert is easier to learn from than a broad feature with unsettled product decisions.
1. Choose one repeatable job
Choose work that already matters, is narrow enough to check, common enough to repeat, and safe enough to recover when the system makes a mistake. Good candidates include documentation corrections, narrow bug fixes, and test-backed migrations.
Write the outcome in product language. “Fix expired discount codes” becomes “Keep the cart, explain why the code failed, never charge unexpectedly, and preserve existing checkout behavior.”
2. Create the task brief
The task brief is the specification and definition of done. It should name:
- the user or system outcome;
- what the agent may and may not change;
- known product decisions and open questions;
- risk, owner, and required approvals;
- checks that prove the result;
- how to release and recover.
If the agent would have to invent a pricing, legal, security, or experience decision, the brief is not ready.
3. Record today's baseline
Run several representative examples through the current process. Measure delivery time, human effort, rework, failures, recovery, and total cost. Keep the examples. A pilot needs a comparison with real work, not a polished demo.
4. Prepare the agent harness
The agent harness should provide the right repository state, tools, supplied context, and a sandbox with limited permissions. Pin important tool versions. Control credentials and network access. Keep a run record of what the agent saw, changed, called, and checked.
Use ordinary software for exact steps. Use a fixed AI workflow when the path is known but interpretation helps. Use an adaptive coding agent when the path is uncertain and the outcome can still be verified. Anthropic's practitioner guide recommends adding agentic complexity only when it improves the result.
5. Build the proof loop and human approval gates
Create realistic eval cases before expanding autonomy. Include common examples, important edge cases, known past failures, and at least one case that should make the agent stop.
Combine deterministic tests with human judgment where product taste or ambiguity matters. Preserve an evidence package containing the brief, relevant context, run record, exact artifact, checks, approvals, release, and production result.
Name every human approval gate, which SWFT calls an attention gate. A person might approve a product choice, sensitive data access, security exception, merge, or production release. Give the run a time and cost budget, explicit write boundaries, stop conditions, and a recoverable path for external actions.
6. Run the pilot and improve one boundary at a time
Review accepted and rejected runs together. Classify repeated problems by the part of the system that should improve:
- unclear brief;
- missing or stale context;
- model limitation;
- poor tool;
- unreliable workspace;
- weak eval or test;
- permission conflict;
- release failure;
- wrong product assumption.
Fix the reusable boundary and add the failure to the eval suite. Correcting one patch is recovery. Preventing the same class of failure is learning.
Expand the task scope, permissions, parallel agents, or proactive triggers one dimension at a time. METR's time-horizon research is a reminder that autonomy is probabilistic. State the kind of work and reliability you measured.
Sample pilot brief
| Field | Example |
|---|---|
| Goal | Explain expired discount codes without losing the cart |
| Allowed scope | Checkout error handling, copy, and tests |
| Must not change | Pricing, payment capture, or discount eligibility |
| Proof | Regression tests, checkout scenario, screenshot, human copy review |
| Human approval gates | Any pricing ambiguity, security finding, merge, production release |
| Recovery | Revert the exact released change and preserve the run record |
Minimum launch checklist
- One recurring job with an owner
- A task brief with a clear definition of done
- Real baseline examples and measurements
- A safe, reproducible agent harness
- Declared context and permissions
- Realistic evals and a complete evidence package
- Named human approval gates, budgets, and stop rules
- A checked release and recovery path
- A weekly review of results, quality, attention, cost, and learning
The first production line is successful when the next valid request follows the same understandable path and a failure leaves enough evidence to improve that path.