GameHorizon Suite: Multi-Horizon Data and Evaluation in Gameplay
GameHorizon pairs 5,000 hours of AAA gameplay data with offline and online multi-horizon model tests.
GameHorizon is a data and evaluation suite for gameplay across short and long temporal horizons. Its annotator builds multi-horizon instructions, and GameHorizon-Data contains 5,000 hours of aligned video, actions, and instructions from 21 AAA games recorded by 100 expert players. GameHorizon-Bench adds reproducible offline questions and stepwise online tests. Evaluation of 47 models through more than one million invocations shows a clear task-difficulty hierarchy and large capability gaps. The dataset, annotator, and benchmark are planned for release.
- GameHorizon-Data holds 5,000 hours from 21 games by 100 expert players.
- An automated annotator produces multi-horizon language instructions.
- Offline tests use thousands of standardized questions across three tasks.
- Online stepwise tests check whether offline scores predict real play.
- Forty-seven models were evaluated with over one million invocations.
Full article238 words · extracted from huggingface.co · click to collapse
Modern video games provide a measurable testbed for AI models, combining abilities of visual understanding, instruction decomposition, goal planning, and precise action control over multiple temporal horizons. Existing datasets and benchmarks, however, either cover a narrow range of games, lack language instructions, or rely on high-variance online rollouts. To address these challenges, we introduce GameHorizon, a unified data and evaluation suite that measures gameplay capabilities at different horizons for diverse model families. GameHorizon Suite consists of three components. First, GameHorizon-Annotator is a scalable and automated annotation pipeline for multi-horizon instructions. Second, utilizing the pipeline, we construct GameHorizon-Data, the first large-scale AAA gameplay dataset with temporally aligned videos, player actions, and multi-horizon instructions. It comprises 5,000 hours of recordings from 21 games, collected by 100 human expert players. Third, we build GameHorizon-Bench with reproducible offline and stepwise online testing. The offline track enables reproducible evaluation using thousands of standardized questions organized into three primary tasks and a series of diagnostic variants, while the online track tests whether offline scores reflect actual gameplay capabilities and localizes failures to specific steps within long-horizon gameplay. Based on our GameHorizon Suite, we evaluate 47 models through more than one million model invocations, revealing a meaningful hierarchy of task difficulty and pronounced differences in model capabilities. Our work can provide a standardized yardstick for evaluating gameplay capabilities across horizons and model families. We will release our dataset, annotator, and benchmark to facilitate future research.
Text extracted automatically; images, tables and formatting may be missing. Original: https://huggingface.co/papers/2609.25001