GlobeBench measures LLM text world-model simulation accuracy
Original title: GlobeBench: Evaluating Text-Based World Model Capabilities for Scalable Environment Simulations
Across 1,632 tasks, GPT-6 Astra leads 20 models at 49.57%, and 631 tasks stay unsolved.
A new paper introduces GlobeBench to compare text-based world models that predict how an environment responds to an action given the full agent trajectory. The benchmark has 1,632 tasks across five domains and nineteen simulation skills, scored against an execution-derived reference with deterministic gates and a semantic judge. Among 20 general-purpose LLMs and specialized text world models, GPT-6 Astra has the highest task success rate at 49.57%. Accuracy falls when changes must propagate to dependent objects, to an average of 5.1%.
- Tasks span five domains and nineteen simulation skills.
- Success is judged against an execution-derived reference.
- Dependent-object propagation drops average success to 5.1%.
- 631 tasks remain unsolved by any of the 20 models.
From the original article
1 min read at huggingface.co
Text-based world models have recently shown potential to simulate complex RL environment state transitions for a diverse set of use cases. However, no standardized benchmark exists to compare, study and improve these models. Through this paper, we attempt to solve this problem by building GlobeBench, a novel benchmark with 1,632 tasks across five separate domains and nineteen simulation skills.
By Nicholas Saban, Charvi Bannur, Mariya Vasileva · huggingface.co
A short excerpt; the article and its images belong to the publisher. Summary, key facts and assessment above are AI-written from the article and can be wrong.