LLM world models pass mechanical checks but miss intended structure
Original title: Evaluating LLM-Generated World Models for Neuro-Symbolic Reasoning
Of 134 valid game-track models, mean structural F1 is 0.366 and only 28 exactly match the reference.
The paper tests whether five LLMs can formalize open-ended situations as partially observable stochastic games. A 44-scenario benchmark spans game, situational, and cultural settings, and 410 first-pass generations are scored for mechanical validity, structural agreement, and semantics. In the reference-backed game track, 134 of 135 models pass a strict mechanical audit, yet mean structural F1 is 0.366 and only 28 of 134 match the reference exactly. Naming a country changes structure for most models while barely changing automated semantic scores.
- The benchmark contains 44 game, situational, and cultural scenarios.
- Five LLMs produced 410 first-pass world-model generations.
- In the game track, 134 of 135 models passed the mechanical audit.
- Only 28 of 134 valid models exactly matched the authored reference.
- Naming the country changes structure for most LLMs in the cultural track.
From the original article
1 min read at arxiv.orgFormal world models can support planning, verification, and safety reasoning, but constructing them from open-ended natural-language situations is difficult, particularly in multi-agent settings where several formalizations may be plausible.
By Souhaila Djaffal · arxiv.org
A short excerpt; the article and its images belong to the publisher. Summary, key facts and assessment above are AI-written from the article and can be wrong.