--- schema: "swft.publication/v1" id: "harnesstax-coding-agent-cost" title: "The coding-agent harness is a cost lever" description: "HarnessTax finds coding-agent harness choice can change token cost far more than success on SWE-bench Lite and Terminal-Bench 2.0. Factory leaders should compare harnesses, not inherit the provider default." summary: "A Berkeley–Arena study of seven models in Claude Code, Codex CLI, and Pi found similar success rates at substantially different token costs. A minimal open harness was competitive. Models often did better outside the provider’s own wrapper. That is evidence about two public benches and token prices, not a factory-wide tax or a production recommendation." canonical: "https://swft.io/ideas/harnesstax-coding-agent-cost" author: "SWFT Editorial" author_type: "Organization" published: "2026-09-17" modified: "2026-09-17" kind: "analysis" section: "Ideas" tags: ["HarnessTax", "agent harness", "coding agent cost", "software factory metrics", "SWE-bench", "Terminal-Bench"] evidence_labels: ["INFERENCE", "STUDY"] source_ids: ["harnesstax-pan-2026", "pi-coding-agent", "swe-bench-lite", "terminal-bench-2"] authorship_disclosure: "AI-drafted from the cited public sources and independently checked by a second AI editorial-review agent (Codex) for source fit, claim boundaries, overlap, and reader utility. SWFT Editorial is responsible for corrections." --- # The coding-agent harness is a cost lever HarnessTax compared 21 model–harness pairs on two public coding-agent benches. Harness choice moved token cost far more than success. Do not inherit a provider default without a comparison. > **Authorship:** AI-drafted from the cited public sources and independently checked by a second AI editorial-review agent (Codex) for source fit, claim boundaries, overlap, and reader utility. SWFT Editorial is responsible for corrections. The wrapper around a coding model can change what you pay more than whether the task succeeds. [HarnessTax](https://harnesstax.github.io/), a September 2026 study by Melissa Z. Pan, Shuo Yang, Negar Arabzadeh, Wei-Lin Chiang, Ion Stoica, and Matei Zaharia, makes that claim on two public coding-agent benches. For a software factory, the useful reading is narrower than a model bake-off: treat the **agent harness** as a cost and control decision, not a loyalty test. An agent harness is the software around the model: instructions, tools, context, permissions, workspace, and feedback. Product teams often inherit the provider default—Claude Code with Anthropic models, Codex CLI with OpenAI models—because it is the path of least resistance. HarnessTax’s evidence is that this inheritance can hide a large token bill for little extra success on the work it tested. ## What HarnessTax measured The authors evaluated 21 model–harness pairs: seven models in three harnesses, Claude Code, Codex CLI, and Pi, on [SWE-bench Lite](https://www.swebench.com/lite) and [Terminal-Bench 2.0](https://arxiv.org/abs/2601.11868). They used the same 30 randomly sampled tasks from each benchmark, ran each pair three times per task, started from each harness’s native high-effort setting, and capped an attempt at 100 agent turns. Success used each benchmark’s official evaluator. Token cost used a fixed direct-API price list dated 1 September 2026, applied to the same model in every harness. That design answers a comparative question: holding the model and the task sample fixed, what does the harness change? It does not measure a production factory. The tasks are public benchmark items. The authors note that models may have seen these benches during training, and that results may differ on other workloads. Turn counts follow each harness’s own definition. On SWE-bench Lite they blocked container network access and disabled the default web tools in Claude Code and Codex. ## Cost moved more than success The study’s first finding is that harness choice had little effect on success and a large effect on cost. The authors state that the same model can keep a similar success rate at up to about 5× the token cost. Across shared models they report geometric-mean cost ratios: Claude Code cost about 2.0× Pi and 1.6× Codex on SWE-bench Lite, and about 1.5× Pi on Terminal-Bench 2.0. The average harness effect on success stayed within ±2% on SWE-bench Lite and about ±5% on Terminal-Bench 2.0. A concrete pair: Claude Fable 5 solved 97.8% of attempts in Claude Code, 96.7% in Codex, and 96.7% in Pi. Claude Code cost about twice Pi on that comparison, $1.33 versus $0.67 per the authors’ price list. Fable 5’s extra success in Claude Code was 1.1 percentage points at about twice the spend, while mean recorded turns were nearly identical (15.3 versus 15.4). The extra money was not extra looping in the sense the authors could count. The authors call that gap a **harness tax**: paying more for essentially the same quality because a default wrapper was accepted without a comparison. Keep the unit visible. This is token cost on two benches, not review time, incident cost, or accepted factory output. ## A simple open harness can compete Pi reached the Pareto frontier on both benches with four tools: read, write, edit, and bash. The authors present that as evidence that a minimal, inspectable harness can be competitive on cost and success in this setting. They also locate part of the tax in the first model call. Across all seven models, Claude Code’s mean initial context was more than 10× Pi’s, with longer instructions and larger tool schemas. More context can raise cost. Total spend still depends on caching, generated tokens, and later calls. The authors treat harness complexity as an empirical trade-off: richer features may help other models, workloads, or interactive settings. They do not claim that every production factory should strip tools until four remain. The factory reading is operational. If you cannot inspect what the harness injects, you cannot tell whether you are paying for useful context or for a standing prompt and tool catalog. A small open harness is a research and comparison option. It is not, on this evidence, a drop-in replacement for a provider environment with your permissions, secrets, and review path. ## The provider default is not automatically best Providers sometimes optimize a model for their own coding environment. The authors note that fact and then report the opposite pairing result. Across the six Anthropic and OpenAI models and both benches, an alternative harness had the highest observed success in nine of twelve comparisons. Two examples they give: Claude Sonnet 4.6 solved 68.9% of attempts in Codex versus 66.7% in Claude Code on SWE-bench Lite at a similar cost. GPT-5.6 Sol on Terminal-Bench 2.0 reached 83.3% in Pi versus 78.9% in Codex at about half the cost, $0.42 versus $0.76. The authors’ conclusion is that a shared provider does not guarantee the best pairing. The practical question is which harness gives the better cost–success balance for a given model and workload. For a factory leader, that undercuts a common procurement shortcut: buy the model and take the vendor’s wrapper as the optimized stack. The study does not say the vendor harness is worse in your repository. It says you should not assume it is better. ## What the study does not establish HarnessTax is one disclosed comparison on two open-source benches, 30 tasks each, three attempts, and a 1 September 2026 price list. It does not measure: - accepted product work, review effort, or escaped defects in a company factory; - multi-session work with developer feedback, which the authors name as the next evaluation step; - whether extra context or tools would help a different task class; - total factory cost, only token cost under a fixed API price list. A cheaper harness that fails your proof loop, leaks data, or creates more senior review is not a saving. SWFT’s [metrics guide](/software-factory-metrics) keeps token spend in the numerator of cost per accepted result. It is not the whole numerator. ## What a factory team should do with this Use the study as a decision prompt, not a ranking of products. 1. Name the harness as an explicit choice beside the model. Record instructions, tools, context budget, and turn or spend caps the way you would record a model version. 2. Compare at least one alternative on a fixed task sample from *your* line, including a simpler harness if you can run it safely. Hold the model and the acceptance rule constant. 3. Judge the comparison on accepted results and total cost, not success rate alone. If two harnesses finish the same jobs, prefer the cheaper inspectable one until a richer wrapper earns its keep. 4. Do not treat “the model’s own harness” as a quality guarantee. Ask which pairing actually won on the work you run. This sits next to SWFT’s [architecture](/software-factory-architecture), where the harness is the layer that lets an agent act, and next to the [three factory visions](/ideas/three-visions-of-the-ai-software-factory), which start with one bounded line. HarnessTax does not choose among those visions. It says that even inside one line, the wrapper is a cost lever you can measure. The authors will release profiling traces and argue for evaluating and eventually automating harness selection in real workflows. Until a team has that measurement on its own work, the cheap decision is the inherited default. The study’s warning is that this cheap decision can be expensive. ## How we know - **Independent study (STUDY)** HarnessTax evaluated 21 model–harness pairs (seven models in Claude Code, Codex CLI, and Pi) on 30 randomly sampled tasks from SWE-bench Lite and from Terminal-Bench 2.0, with three attempts, a 100-turn cap, official evaluators, and a 1 September 2026 direct-API price list applied uniformly across harnesses. Sources: [HarnessTax: How Much Does the Harness Matter for Coding Agents?](https://harnesstax.github.io/); [SWE-bench Lite](https://www.swebench.com/lite); [Terminal-Bench: Benchmarking Agents on Hard, Realistic Tasks in Command Line Interfaces](https://arxiv.org/abs/2601.11868). - **Independent study (STUDY)** The authors report that harness choice changed success little (within about ±2% on SWE-bench Lite and ±5% on Terminal-Bench 2.0) while the same model could keep similar success at up to about 5× token cost; Claude Code cost about 2.0× Pi and 1.6× Codex on SWE-bench Lite and about 1.5× Pi on Terminal-Bench 2.0 by geometric-mean cost ratios. Sources: [HarnessTax: How Much Does the Harness Matter for Coding Agents?](https://harnesstax.github.io/). - **Independent study (STUDY)** Claude Fable 5 solved 97.8% of attempts in Claude Code versus 96.7% in Codex and Pi, at about twice Pi’s token cost ($1.33 vs $0.67) with nearly identical mean recorded turns (15.3 vs 15.4) on SWE-bench Lite. Sources: [HarnessTax: How Much Does the Harness Matter for Coding Agents?](https://harnesstax.github.io/). - **Independent study (STUDY)** Pi reached the Pareto frontier on both benches with four tools (read, write, edit, bash). Across all seven models, Claude Code’s mean initial context was more than 10× Pi’s. Sources: [HarnessTax: How Much Does the Harness Matter for Coding Agents?](https://harnesstax.github.io/); [Pi coding agent](https://github.com/earendil-works/pi/blob/main/packages/coding-agent/README.md). - **Independent study (STUDY)** Across six Anthropic and OpenAI models and both benches, an alternative harness had the highest observed success in nine of twelve comparisons, including Sonnet 4.6 on SWE-bench Lite (68.9% Codex vs 66.7% Claude Code) and GPT-5.6 Sol on Terminal-Bench 2.0 (83.3% Pi vs 78.9% Codex at about half the cost). Sources: [HarnessTax: How Much Does the Harness Matter for Coding Agents?](https://harnesstax.github.io/). - **Analysis (INFERENCE)** SWFT reads HarnessTax as a factory cost-and-comparison prompt: name the harness, compare it on the team’s own accepted-result scorecard, and do not treat the provider default as automatically optimal. The study does not establish production quality, review cost, or a recommendation to adopt Pi. Sources: [HarnessTax: How Much Does the Harness Matter for Coding Agents?](https://harnesstax.github.io/). ## Sources - **Independent study (STUDY)** [HarnessTax: How Much Does the Harness Matter for Coding Agents?](https://harnesstax.github.io/) — HarnessTax; published 2026-09-16; accessed 2026-09-17. Pan, Yang, Arabzadeh, Chiang, Stoica, and Zaharia compare 21 model–harness pairs on SWE-bench Lite and Terminal-Bench 2.0. The project page was generated 16 September 2026 UTC; SWFT accessed the rendered post on 17 September 2026. - **Observed artifact (OBS)** [Pi coding agent](https://github.com/earendil-works/pi/blob/main/packages/coding-agent/README.md) — earendil-works; accessed 2026-09-17. The inspectable README HarnessTax cites for Pi's four-tool interface: read, write, edit, and bash. - **Observed artifact (OBS)** [SWE-bench Lite](https://www.swebench.com/lite) — SWE-bench; accessed 2026-09-17. Official description of the SWE-bench Lite split used as one of HarnessTax's two evaluation suites. - **Independent study (STUDY)** [Terminal-Bench: Benchmarking Agents on Hard, Realistic Tasks in Command Line Interfaces](https://arxiv.org/abs/2601.11868) — arXiv; published 2026-01-17; accessed 2026-09-17. Merrill et al., arXiv:2601.11868, the Terminal-Bench paper HarnessTax cites for Terminal-Bench 2.0. ## Read next - [Software factory metrics that matter](/software-factory-metrics) - [How an AI software factory works](/software-factory-architecture) - [Three visions of the AI software factory](/ideas/three-visions-of-the-ai-software-factory)