Continued pre-training study scales embodied agents across virtual worlds
Original title: Scaling in Games: Continued Pre-Training For Embodied Agents In Diverse Virtual Worlds
One hundred twenty runs span 0.8B–27B models, 50–1,600 hours of data, and five world mixtures.
A controlled study tests whether scaling data, diversity, and model size helps embodied agents when virtual worlds differ in visuals, dynamics, controls, and goals. One hundred twenty continued pre-training configurations cover four sizes from 0.8B to 27B, six budgets from 50 to 1,600 hours, and mixtures from Minecraft-only to multi-world data. More data helps in-corpus play, diversity helps related worlds, and larger models are more data-efficient but transfer worse to distant worlds under single-world training. The authors say models match Minecraft specialists and frontier models in 3D worlds but still lag in distant worlds.
Unconfirmed: Comparisons with frontier models are the authors' gameplay benchmarks.
- Model sizes range from 0.8B to 27B parameters.
- Data budgets run from 50 to 1,600 hours.
- An early interface phase gives broad cross-world gains; a later phase narrows them.
- Authors fit a scaling law on data volume, diversity, and model size.
- They argue imitation and continued pre-training alone are not enough for generalists.
From the original article
1 min read at huggingface.co
Scaling data, diversity and model capacity has improved embodied learning in physical environments, where diverse experiences remain grounded in shared physical laws. Whether these gains extend to virtual worlds with different visual styles, dynamics, controls, and objectives remains unclear.
By Kuan Zhang, Yukun Chen, Zhihao Yang · huggingface.co
A short excerpt; the article and its images belong to the publisher. Summary, key facts and assessment above are AI-written from the article and can be wrong.