Grid-Bench: Can LLMs Predict How Discrete Worlds Change?
Grid-Bench tests whether LLMs can predict full grid states after sequences of spatial operations.
Grid-Bench is an executable benchmark of discrete two-, three-, and four-dimensional grids changed by eight families of spatial operations. Given an initial grid, explicit rules, and a text event history, a model must predict the complete final grid, which reference execution can check cell by cell. Six LLMs were evaluated. Complete-state accuracy drops sharply as histories lengthen, and models with similar exact-match scores still differ substantially in how many cells they get wrong.
- Tasks cover 2D, 3D, and 4D grids and eight operation families.
- Models must predict the full final grid from text histories.
- Complete-state accuracy falls sharply as histories grow longer.
- Similar exact-match scores can hide large differences in wrong cells.
From the original article
1 min read at arxiv.orgSpatial instructions describe how to move objects, transform regions, or modify their surroundings. When several operations are applied in sequence, each acts on a configuration shaped by earlier changes. How accurately can large language models (LLMs) predict the resulting spatial state?
By Zixiang Xu · arxiv.org
A short excerpt; the article and its images belong to the publisher. Summary, key facts and assessment above are AI-written from the article and can be wrong.