AutoDataBench: Can Agents Write the Data That Feeds the Self-Improvement Loop?
AutoDataBench finds no agent scores above 20 of 100 writing accepted training tasks in 45 minutes.
AutoDataBench evaluates whether agents can write individual training tasks that meet production acceptance criteria, rather than judging them only after a later training run. Given an original benchmark task and a target model's attempt, the agent must produce a new task for the same suite that is valid, novel, difficult, and behaviorally covering. Across three executable-agent benchmarks, no evaluated agent scores above 20 of 100 at the default 45-minute budget. Quadrupling time substantially improves the strongest agent, while the cost of one usable task stays nearly flat. Code and data are released.
- Agents must write one new executable task meeting validity, novelty, difficulty, and coverage criteria.
- No evaluated agent scores above 20 of 100 within a 45-minute budget.
- Four times more time substantially raises the strongest agent's score.
- Cost per usable task stays almost unchanged despite the longer budget.
- The benchmark measures autonomous data synthesis without running a training loop.
- Products
- AutoDataBench
From the original article
1 min read at huggingface.co
Recent gains in language model capability have come more from data than from architecture. Frontier labs and data companies produce verifiable agentic tasks, which supervised finetuning and reinforcement learning then turn into capability. This production line still rests on human labour and on human-in-the-loop collaboration.
By Haotian Luo, Haoyu Wang, Zeyu Qin · huggingface.co
A short excerpt; the article and its images belong to the publisher. Summary, key facts and assessment above are AI-written from the article and can be wrong.