Auto-Research Arena benchmarks autonomous language-model research
Original title: Auto-Research Arena: Benchmarking and Advancing Auto-Research in Language Modeling
A sequential search baseline beat several complex harnesses; reused ideas lowered loss under matched FLOPs.
The paper introduces Auto-Research Arena, a multi-target benchmark that separates admissibility from improvement when scoring autonomous research systems. For language-model pretraining it uses nine targets across model, data, training and inference efficiency and evaluates seven harnesses with two foundation models. Several sophisticated methods were outperformed by a simple sequential search baseline. Reusing discoveries in a quality-driven search produced lower training loss under matched data and FLOPs and lower held-out BPB at the same model size. Code is on GitHub.
- Nine targets cover model, data, training and inference efficiency.
- Seven research harnesses were tested with two foundation models.
- A sequential search baseline beat several more complex methods.
- Code is published at Auto-Research-Arena/auto-research-arena.
From the original article
1 min read at arxiv.orgAs autonomous research systems become more capable, benchmarks must both evaluate their performance and support meaningful research progress. We introduce Auto-Research Arena, a multi-target benchmark for evaluating autonomous research systems and discovering ideas in language modeling.
By Xi Chen · arxiv.org
A short excerpt; the article and its images belong to the publisher. Summary, key facts and assessment above are AI-written from the article and can be wrong.