---
schema: "swft.publication/v1"
id: "does-ai-coding-work"
title: "Does AI coding actually work? What the evidence shows"
description: "The evidence on AI coding splits by type: METR's RCT found a 19% slowdown with early-2025 tools, DORA finds modest amplifier effects, and operators report large merged-work shares. The denominators differ."
summary: "The evidence splits by type: METR's controlled trial found a 19% slowdown with early-2025 tools (and a weak-signal reversal in its 2026 follow-on), DORA finds modest amplifier effects, and operators report large merged-work shares inside prepared systems. The contradiction is mostly different denominators."
canonical: "https://swft.io/ideas/does-ai-coding-actually-work"
author: "SWFT Editorial"
author_type: "Organization"
published: "2026-09-23"
modified: "2026-09-23"
kind: "analysis"
section: "Ideas"
tags: ["does AI coding work", "AI coding productivity", "METR study", "DORA report", "agentic coding evidence"]
evidence_labels: ["INFERENCE", "OBS", "SELF-REPORT", "STUDY"]
source_ids: ["dora-2025", "dora-2025-errata", "harnesstax-pan-2026", "metr-early-2025-rct", "metr-uplift-update-2026", "ramp-swe-bench", "spotify-code-no-longer-constraint", "stripe-minions-scale-update", "swe-bench-lite", "terminal-bench-2", "uber-efficient-software-factory"]
authorship_disclosure: "AI-drafted from the cited public sources and independently checked by a second AI editorial-review agent (Codex) for source fit, claim boundaries, overlap, and reader utility. SWFT Editorial is responsible for corrections."
---

# Does AI coding actually work? What the evidence shows

A controlled 19% slowdown, industry surveys calling AI an amplifier, and operator reports of thousands of agent PRs a week — reconciled by evidence type.

> **Authorship:** AI-drafted from the cited public sources and independently checked by a second AI editorial-review agent (Codex) for source fit, claim boundaries, overlap, and reader utility. SWFT Editorial is responsible for corrections.

Does AI coding actually work? The honest answer is three honest answers, because different kinds of evidence measure different things. A controlled trial measures what happened to specific developers. An industry survey measures what organizations associate with adoption. An operator report measures what one company says about itself. Read across all three and a usable picture emerges — narrower than the hype, larger than the dismissal.

## What the controlled study found

[METR's July 2025 randomized trial](https://metr.org/blog/2025-07-10-early-2025-ai-experienced-os-dev-study/) assigned 246 real issues across 16 experienced open-source developers to AI-allowed or AI-disallowed work. With early-2025 tools, the AI-allowed tasks took **19 percent longer** — and the developers believed they had been about 20 percent faster. Perception and measurement pointed in opposite directions.

[METR's February 2026 update](https://metr.org/blog/2026-02-24-uplift-update/) complicates the clean story in both directions. The follow-on estimates an 18 percent speedup for returning developers and 4 percent for new ones — but participants had begun refusing AI-free assignments, so METR itself calls the signal weak and is redesigning the study. The fairest reading: the early-2025 slowdown was real in its setting, the effect has likely moved since, and nobody has a clean current measurement.

## What the industry survey found

[DORA's 2025 report](https://dora.dev/research/2025/dora-report/) describes AI as an amplifier: adoption is broad, AI use correlates with modest gains in throughput, and the same organizations report low trust in generated work alongside continued instability. The survey measures associations across thousands of respondents — direction, not controlled effects — and DORA maintains a public [errata record](https://dora.dev/research/2025/errata/) for it.

## What operators report at scale

The strongest claims come from organizations reporting on themselves. Stripe's president reported [roughly 7,000 pull requests from Minions in one week](https://www.youtube.com/watch?v=P5iICDVn5gc) — about 30 percent of that week's PRs. [Uber's factory account](https://www.uber.com/us/en/blog/efficient-software-factory/) reports token economics and adoption across layers. [Spotify's account](https://engineering.atspotify.com/2026/6/code-with-claude-coding-is-no-longer-the-constraint) says coding is no longer the constraint — and that review, standardization, and context became the new ones. [Ramp's SWE-Bench](https://labs.ramp.com/swebench) rebuilt the benchmark from production tasks because public ones did not match its work.

These are first-party reports: real evidence with a known bias toward the flattering number. They are also, at present, the only org-level scale data that exists.

## How to read the contradiction

The disagreement is smaller than it looks because the denominators differ:

- **Benchmarks** ([SWE-bench Lite](https://www.swebench.com/lite), [Terminal-Bench 2.0](https://arxiv.org/abs/2601.11868), [HarnessTax](https://harnesstax.github.io/)) measure capability on bounded tasks — what an agent *can* do, not what a team *gets*.
- **The METR trial** measured individual developers on repositories they knew intimately, with tools from early 2025 — before current harnesses and models.
- **Operator reports** measure merged-work share inside prepared systems: task briefs, scoped context, checks, and human review. The system, not the model alone, produces the number.

A working rule: trust labeled evidence over averages, and treat "does it work" as a question about a specific task class inside a specific system. The figures that will actually decide your next quarter are the ones you measure — [the metrics guide](/software-factory-metrics) shows how to instrument a bounded pilot so your evidence, not the discourse, answers the question.

## How we know

- **Independent study (STUDY)** METR's RCT measured a 19% slowdown on real issues for experienced open-source developers using early-2025 AI tools, with a large perception gap; its 2026 follow-on shows speedup estimates but with selection effects METR flags as weakening the signal. Sources: [Measuring the Impact of Early-2025 AI on Experienced Open-Source Developer Productivity](https://metr.org/blog/2025-07-10-early-2025-ai-experienced-os-dev-study/); [We are Changing our Developer Productivity Experiment Design](https://metr.org/blog/2026-02-24-uplift-update/).
- **Independent study (STUDY)** DORA's 2025 report associates AI adoption with modest throughput gains and describes AI as an amplifier, with low trust in generated work. Sources: [State of AI-assisted Software Development 2025](https://dora.dev/research/2025/dora-report/); [DORA Research: 2025 Errata](https://dora.dev/research/2025/errata/).
- **First-party report (SELF-REPORT)** Operators report large merged-work shares: ~7,000 Minions PRs in a Stripe week, Uber's factory adoption and token economics, and Spotify's shift of the constraint from coding to review. Sources: [Tokens Are the New Dollars](https://www.youtube.com/watch?v=P5iICDVn5gc); [Running a Software Factory Efficiently at Uber Scale](https://www.uber.com/us/en/blog/efficient-software-factory/); [Code with Claude: coding is no longer the constraint](https://engineering.atspotify.com/2026/6/code-with-claude-coding-is-no-longer-the-constraint).
- **Observed artifact (OBS)** Public and production-grounded benchmarks bound what agent capability scores can establish about delivered productivity. Sources: [SWE-bench Lite](https://www.swebench.com/lite); [Terminal-Bench: Benchmarking Agents on Hard, Realistic Tasks in Command Line Interfaces](https://arxiv.org/abs/2601.11868); [HarnessTax: How Much Does the Harness Matter for Coding Agents?](https://harnesstax.github.io/); [Ramp SWE-Bench](https://labs.ramp.com/swebench).
- **Analysis (INFERENCE)** The apparent contradiction between a measured individual slowdown and reported organizational scale is largely denominators: individual issue time on familiar repositories versus merged-work share inside prepared factory systems. Sources: [Measuring the Impact of Early-2025 AI on Experienced Open-Source Developer Productivity](https://metr.org/blog/2025-07-10-early-2025-ai-experienced-os-dev-study/); [State of AI-assisted Software Development 2025](https://dora.dev/research/2025/dora-report/); [Tokens Are the New Dollars](https://www.youtube.com/watch?v=P5iICDVn5gc); [Running a Software Factory Efficiently at Uber Scale](https://www.uber.com/us/en/blog/efficient-software-factory/).

## Sources

- **Independent study (STUDY)** [State of AI-assisted Software Development 2025](https://dora.dev/research/2025/dora-report/) — DORA; accessed 2026-08-31. DORA's research report on AI adoption, throughput, stability, platform quality, and organizational conditions.
- **Observed artifact (OBS)** [DORA Research: 2025 Errata](https://dora.dev/research/2025/errata/) — DORA; updated 2025-11-24; accessed 2026-08-31. The official correction record for the 2025 report and its current versioning practice.
- **Independent study (STUDY)** [HarnessTax: How Much Does the Harness Matter for Coding Agents?](https://harnesstax.github.io/) — HarnessTax; published 2026-09-16; accessed 2026-09-17. Pan, Yang, Arabzadeh, Chiang, Stoica, and Zaharia compare 21 model–harness pairs on SWE-bench Lite and Terminal-Bench 2.0. The project page was generated 16 September 2026 UTC; SWFT accessed the rendered post on 17 September 2026.
- **Independent study (STUDY)** [Measuring the Impact of Early-2025 AI on Experienced Open-Source Developer Productivity](https://metr.org/blog/2025-07-10-early-2025-ai-experienced-os-dev-study/) — METR; published 2025-07-10; accessed 2026-09-23. Becker, Rush, Barnes, and Rein's randomized controlled trial: 16 experienced open-source developers, 246 real issues; AI-allowed tasks took 19% longer with early-2025 tools while developers believed they were faster. METR flags the results as superseded by its late-2025 follow-on.
- **Independent study (STUDY)** [We are Changing our Developer Productivity Experiment Design](https://metr.org/blog/2026-02-24-uplift-update/) — METR; published 2026-02-24; accessed 2026-09-23. METR's late-2025 follow-on: the panel subset estimates -18% speedup (CI -38% to +9%) and new developers -4% (CI -15% to +9%), but severe selection effects — participants refusing AI-free assignments — make it weak evidence; METR is redesigning the study.
- **First-party report (SELF-REPORT)** [Ramp SWE-Bench](https://labs.ramp.com/swebench) — Ramp Labs; updated 2026-06-11; accessed 2026-09-01. Ramp's production-grounded private benchmark of 80 reconstructed tasks, one-attempt scoring, test-based acceptance, and human curation.
- **First-party report (SELF-REPORT)** [Code with Claude: coding is no longer the constraint](https://engineering.atspotify.com/2026/6/code-with-claude-coding-is-no-longer-the-constraint) — Spotify Engineering; published 2026-06-03; accessed 2026-09-01. Spotify's current adoption account and its warning that review, standardization, context, and organizational decisions become the constraints as code generation accelerates.
- **First-party report (SELF-REPORT)** [Tokens Are the New Dollars](https://www.youtube.com/watch?v=P5iICDVn5gc) — Andreessen Horowitz; published 2026-08-17; accessed 2026-09-01. Stripe president Will Gaybrick reports that roughly 7,000 PRs came from Minions in the preceding week, about 30 percent of that week's PRs; he does not say all were merged.
- **Observed artifact (OBS)** [SWE-bench Lite](https://www.swebench.com/lite) — SWE-bench; accessed 2026-09-17. Official description of the SWE-bench Lite split used as one of HarnessTax's two evaluation suites.
- **Independent study (STUDY)** [Terminal-Bench: Benchmarking Agents on Hard, Realistic Tasks in Command Line Interfaces](https://arxiv.org/abs/2601.11868) — arXiv; published 2026-01-17; accessed 2026-09-17. Merrill et al., arXiv:2601.11868, the Terminal-Bench paper HarnessTax cites for Terminal-Bench 2.0.
- **First-party report (SELF-REPORT)** [Running a Software Factory Efficiently at Uber Scale](https://www.uber.com/us/en/blog/efficient-software-factory/) — Uber Engineering; published 2026-08-27; accessed 2026-08-31. Uber's account of factory layers, adoption, token economics, managed agents, quality signals, and context infrastructure.

## Read next

- [Software factory metrics that matter](/software-factory-metrics)
- [Agent swarm scale: documented runs and adoption](/scale)
- [The coding-agent harness is a cost lever](/ideas/harnesstax-coding-agent-cost)