AutoResearch at Production Scale: Failure Modes and a Multi-Agent Framework
Production AutoResearch on book embeddings hit five failure modes; a scaffolding fix lifted Recall@6 1.82x.
The paper applies Andrej Karpathy's AutoResearch loop—an LLM that edits a training script and keeps changes that improve a held-out metric—to production embedding systems for book recommendations. Over twelve weeks and more than 220 experiments on two systems, the authors identify five failure modes: infrastructure fragility, agent memory decay, search-direction stagnation, iteration-cost asymmetry, and metric fixation. A prevent-persist-redirect scaffolding produced a 1.82x Recall@6 lift and a 2.1x coherence lift over hand-tuned baselines, plus an autonomous text-only fallback that expanded catalog coverage 5.8x. The same modes appeared across nearly three orders of magnitude in per-iteration cost.
- AutoResearch loop edits training scripts and keeps changes that raise a held-out metric.
- Twelve weeks and 220+ experiments covered two book-recommendation embedding systems.
- Five failure modes include memory decay, search stagnation, and metric fixation.
- Prevent-persist-redirect scaffolding maps each failure mode to a structural remedy.
- Result: 1.82x Recall@6, 2.1x coherence, and 5.8x catalog coverage via a text-only fallback.
From the original article
1 min read at huggingface.co
Optimizing embedding systems for production recommendation pipelines demands systematic exploration that consumes disproportionate engineering effort at scale. We apply Andrej Karpathy's AutoResearch paradigm -- a large language model that iteratively edits a training script and retains modifications that improve a held-out scalar metric -- to automate this exploration.
By Aparajith Chandran, Juwon Kim, Saurav Jha · huggingface.co
A short excerpt; the article and its images belong to the publisher. Summary, key facts and assessment above are AI-written from the article and can be wrong.