--- schema: "swft.publication/v1" id: "company-airbnb-llm-migration" title: "Airbnb's LLM migration factory: 3,500 test files with a proof loop" description: "How Airbnb migrated nearly 3,500 React tests with per-file states, parallel model work, validation gates, retries, context engineering, and a human finish." summary: "Airbnb split a 3,500-file test migration into independent work units with explicit stages. Models attempted each conversion, established tools checked it, and failures guided another attempt. The pipeline reached 97 percent before engineers used the partial work to complete the difficult final 3 percent." canonical: "https://swft.io/companies/airbnb-llm-migration" author: "SWFT Editorial" author_type: "Organization" published: "2026-09-01" modified: "2026-09-02" kind: "case-study" section: "Companies" tags: ["Airbnb LLM migration", "AI code migration", "React Testing Library migration", "validation state machine", "context engineering", "human escalation"] evidence_labels: ["INFERENCE", "SELF-REPORT"] source_ids: ["airbnb-llm-test-migration"] authorship_disclosure: "AI-drafted from the cited public sources and independently checked by a second AI editorial-review agent (Codex) for source fit, claim boundaries, overlap, and reader utility. SWFT Editorial is responsible for corrections." --- # Airbnb's LLM migration factory: 3,500 test files with a proof loop Airbnb gave every test file a visible state, used ordinary tools to check model-generated conversions, improved common failures in batches, and sent the final 3 percent to people. > **Authorship:** AI-drafted from the cited public sources and independently checked by a second AI editorial-review agent (Codex) for source fit, claim boundaries, overlap, and reader utility. SWFT Editorial is responsible for corrections. ## Quick answer Airbnb split a 3,500-file test migration into independent work units with explicit stages. Models attempted each conversion, established tools checked it, and failures guided another attempt. The pipeline reached 97 percent before engineers used the partial work to complete the difficult final 3 percent. Airbnb turned one large code migration into a measurable assembly line. Each test file moved through explicit validation stages. A model attempted the uncertain conversion, ordinary tools checked the result, and failures became the next attempt's instructions. The system automated 97 percent of nearly 3,500 files before people finished the long tail. ## What was Airbnb's LLM test migration? Airbnb needed to replace Enzyme with React Testing Library across nearly 3,500 React test files. The two tools encourage different testing styles, so a simple text replacement could not preserve the intent of each test. Airbnb estimated that a manual migration would require about 1.5 years of engineering work. Its [March 2025 engineering account](https://medium.com/airbnb-engineering/accelerating-large-scale-test-migration-with-llms-9565c208023b) says the completed project took six weeks. That result combines model calls, migration tooling, prompt and script refinement, and human work. It is a company case study, not a controlled comparison of equivalent teams. ## How did Airbnb turn files into a production pipeline? Airbnb modeled each file as a small state machine. A file moved through a test conversion, Jest repair, lint repair, TypeScript repair, and completion. It advanced only when the validation for the current stage passed. This structure made progress visible and recoverable. The team could rerun one file, one directory, or every file stuck at the same stage. It could also process hundreds of independent files in parallel. The model handled the part that required interpretation. Jest, ESLint, TypeScript, and the test suite acted as deterministic proof. A failed check did not merely mark the work unsuccessful. The system supplied the latest file and validation errors to the next model attempt. ## Why did Airbnb use retries and rich context? Simple and medium-complexity files often completed after several attempts, with most finishing within ten. More complex files needed richer context. By the end of the migration, a prompt could contain 40,000 to 100,000 tokens and as many as 50 related files. The supplied context included the component under test, the test being migrated, current validation errors, nearby tests that showed team conventions, migration guidance, and known good examples. Airbnb says selecting the right related files mattered more than polishing prompt wording. This is context engineering in a concrete form. The team assembled evidence that helped the model make the next specific decision instead of loading unrelated company knowledge. ## How did Airbnb handle the long tail? The first bulk run automatically completed 75 percent of target files in four hours. That still left nearly 900 files. Airbnb stamped each file with machine-readable progress, grouped failures by stage, and used a loop it called sample, tune, and sweep: 1. Run the remaining failures and find a repeated problem. 2. Choose five to ten examples of that problem. 3. Update the prompt or ordinary migration code. 4. Test the change on the sample. 5. Sweep the improved workflow across all remaining files. After four days, Airbnb reported 97 percent completion. Some difficult files had been attempted 50 to 100 times. The team stopped spending model budget at that ceiling and manually completed the final 3 percent from the partial results in another week. The stop decision is part of the factory design. Automation did not need to solve every file to be useful. People took over where repeated attempts stopped producing better results. ## What did Airbnb prove, and what remains uncertain? Airbnb says the migration preserved the original test intent and overall code coverage. It also says the combined model and engineering cost was much lower than the original manual estimate. The article does not disclose model spend, engineer hours by stage, post-migration defects, review burden, or an independent audit of behavioral equivalence. The tooling was built for this migration, and no public repository is linked. The case proves that a carefully bounded transformation can be industrialized. It does not establish a general-purpose internal agent platform or show that the same success rate transfers to feature development. ## What can a product team copy? Choose a repeated change with many independent work units and clear checks. Give every unit a visible state. Use failures as structured feedback. Improve the most common failure class before chasing one exceptional case. Set a retry and cost ceiling, then route the long tail to people with the best partial result attached. Measure completion, elapsed time, model cost, engineering time, manual exceptions, rework, and behavior after release. The useful outcome is a completed safe migration, not the number of model attempts. ## How we know - **First-party report (SELF-REPORT)** Airbnb reports migrating nearly 3,500 files in six weeks against a 1.5-year manual estimate while maintaining original test intent and overall coverage. Sources: [Accelerating large-scale test migration with LLMs](https://medium.com/airbnb-engineering/accelerating-large-scale-test-migration-with-llms-9565c208023b). - **First-party report (SELF-REPORT)** Airbnb describes a per-file validation state machine, parallel work, dynamic retries, and supplied context reaching 40,000 to 100,000 tokens and as many as 50 related files. Sources: [Accelerating large-scale test migration with LLMs](https://medium.com/airbnb-engineering/accelerating-large-scale-test-migration-with-llms-9565c208023b). - **First-party report (SELF-REPORT)** Airbnb reports 75 percent automatic completion in four hours, 97 percent after four days of sample-tune-sweep refinement, and a manual finish for the remaining 3 percent. Sources: [Accelerating large-scale test migration with LLMs](https://medium.com/airbnb-engineering/accelerating-large-scale-test-migration-with-llms-9565c208023b). - **Analysis (INFERENCE)** SWFT treats explicit per-unit state, outside validation, failure-driven retries, and a deliberate human handoff as the reusable factory pattern. Sources: [Accelerating large-scale test migration with LLMs](https://medium.com/airbnb-engineering/accelerating-large-scale-test-migration-with-llms-9565c208023b). ## Sources - **First-party report (SELF-REPORT)** [Accelerating large-scale test migration with LLMs](https://medium.com/airbnb-engineering/accelerating-large-scale-test-migration-with-llms-9565c208023b) — The Airbnb Tech Blog; published 2025-03-13; accessed 2026-09-01. Airbnb's account of a 3,500-file Enzyme-to-React-Testing-Library migration using a per-file state machine, validation gates, bounded retries, rich context, and a manual finish for the final three percent. ## Read next - [Spotify Honk: adding an agent to a software factory that already worked](/companies/spotify-honk) - [Stripe Minions: how developer infrastructure became an AI software factory](/companies/stripe-minions) - [Software factory metrics that matter](/software-factory-metrics)