Scaling Automatic Research Agents via World Models
WMRL replaces environment execution with a world model in RL, accelerating research-agent post-training 3-4x and letting 4B/9B agents beat 48B/120B open-weight agents.
The paper identifies that environment execution dominates RL training cost for automatic research agents because each execution occupies an exclusive sandbox while generation batches efficiently. World Model RL (WMRL) substitutes a learned world model for execution, with Online Debiasing and Inverse-Variance Denoising to handle reward bias and noise, and the paper proves both improve convergence guarantees. WMRL accelerates training 3-4x across tasks and outperforms standard RL baselines; post-trained 4B and 9B agents beat 48B and 120B open-weight agents on held-out benchmarks. WMRL also transfers to post-training embodied VLA policies.
- Environment execution is the RL bottleneck for automatic research agents
- WMRL replaces sandbox execution with a world model plus debiasing and denoising
- Training accelerates 3-4x while exceeding standard RL baselines
- Post-trained 4B and 9B agents outperform 48B and 120B open-weight agents
- Method transfers to embodied VLA policy post-training
From the original article
1 min read at huggingface.co
Automating empirical research is a long-standing direction of AI. Recent automatic research (AutoResearch) agents bring this goal within reach, as modern LLMs show the capability to independently implement solutions and learn from the execution outcomes. Behind these gains, post-training (especially RL) plays a central role.
By Xiyuan Yang, Sheikh Sarwar, Jingru Cheng · huggingface.co
A short excerpt; the article and its images belong to the publisher. Summary, key facts and assessment above are AI-written from the article and can be wrong.