Airbnb turned one large code migration into a measurable assembly line. Each test file moved through explicit validation stages. A model attempted the uncertain conversion, ordinary tools checked the result, and failures became the next attempt's instructions. The system automated 97 percent of nearly 3,500 files before people finished the long tail.
What was Airbnb's LLM test migration?
Airbnb needed to replace Enzyme with React Testing Library across nearly 3,500 React test files. The two tools encourage different testing styles, so a simple text replacement could not preserve the intent of each test. Airbnb estimated that a manual migration would require about 1.5 years of engineering work.
Its March 2025 engineering account says the completed project took six weeks. That result combines model calls, migration tooling, prompt and script refinement, and human work. It is a company case study, not a controlled comparison of equivalent teams.
How did Airbnb turn files into a production pipeline?
Airbnb modeled each file as a small state machine. A file moved through a test conversion, Jest repair, lint repair, TypeScript repair, and completion. It advanced only when the validation for the current stage passed.
This structure made progress visible and recoverable. The team could rerun one file, one directory, or every file stuck at the same stage. It could also process hundreds of independent files in parallel.
The model handled the part that required interpretation. Jest, ESLint, TypeScript, and the test suite acted as deterministic proof. A failed check did not merely mark the work unsuccessful. The system supplied the latest file and validation errors to the next model attempt.
Why did Airbnb use retries and rich context?
Simple and medium-complexity files often completed after several attempts, with most finishing within ten. More complex files needed richer context. By the end of the migration, a prompt could contain 40,000 to 100,000 tokens and as many as 50 related files.
The supplied context included the component under test, the test being migrated, current validation errors, nearby tests that showed team conventions, migration guidance, and known good examples. Airbnb says selecting the right related files mattered more than polishing prompt wording.
This is context engineering in a concrete form. The team assembled evidence that helped the model make the next specific decision instead of loading unrelated company knowledge.
How did Airbnb handle the long tail?
The first bulk run automatically completed 75 percent of target files in four hours. That still left nearly 900 files. Airbnb stamped each file with machine-readable progress, grouped failures by stage, and used a loop it called sample, tune, and sweep:
- Run the remaining failures and find a repeated problem.
- Choose five to ten examples of that problem.
- Update the prompt or ordinary migration code.
- Test the change on the sample.
- Sweep the improved workflow across all remaining files.
After four days, Airbnb reported 97 percent completion. Some difficult files had been attempted 50 to 100 times. The team stopped spending model budget at that ceiling and manually completed the final 3 percent from the partial results in another week.
The stop decision is part of the factory design. Automation did not need to solve every file to be useful. People took over where repeated attempts stopped producing better results.
What did Airbnb prove, and what remains uncertain?
Airbnb says the migration preserved the original test intent and overall code coverage. It also says the combined model and engineering cost was much lower than the original manual estimate. The article does not disclose model spend, engineer hours by stage, post-migration defects, review burden, or an independent audit of behavioral equivalence.
The tooling was built for this migration, and no public repository is linked. The case proves that a carefully bounded transformation can be industrialized. It does not establish a general-purpose internal agent platform or show that the same success rate transfers to feature development.
What can a product team copy?
Choose a repeated change with many independent work units and clear checks. Give every unit a visible state. Use failures as structured feedback. Improve the most common failure class before chasing one exceptional case. Set a retry and cost ceiling, then route the long tail to people with the best partial result attached.
Measure completion, elapsed time, model cost, engineering time, manual exceptions, rework, and behavior after release. The useful outcome is a completed safe migration, not the number of model attempts.