ExplorationBench: Measuring AI Systems' Exploration in Verifiable Alien Worlds
ExplorationBench tests whether AI systems discover new rules in verifiable alien worlds instead of recalling pretraining.
ExplorationBench measures whether AI systems discover new knowledge through exploration rather than recalling pretraining. It uses two verifiable Alien World sandboxes, AlienCode (31 discovery targets, 70 tasks) and AlienLogic (24 discovery targets, 70 tasks), whose executable rules conflict with familiar knowledge. Systems get a flawed manual, environmental feedback, and a tool-call schema, then solve held-out tasks. Across 10 systems, the strongest could apply unfamiliar rules, but results varied and continued exploration sometimes reversed earlier gains.
- AlienCode has 31 discovery targets and 70 tasks; AlienLogic has 24 and 70.
- Executable alien-world rules conflict with familiar knowledge so recall cannot solve tasks.
- Ten AI systems were evaluated; the strongest can acquire unfamiliar rules.
- Performance varies by trajectory, and further exploration can stall or reverse gains.
Full article206 words · extracted from arxiv.org · click to collapse
Scientific discovery begins where known problems end. There, AI systems must engage in exploration: framing hypotheses, designing experiments, and iterating on the results. However, evaluating this ability is difficult: (1) how to verify whether a genuinely new hypothesis holds, and (2) how to determine whether a system has discovered it through exploration or merely recalled related knowledge from pre-training data. To this end, we introduce ExplorationBench, which turns the wicked problem of evaluating scientific exploration into a concrete and tractable framework built on verifiable Alien Worlds: their rules are executable, so every answer can be checked exactly, and they conflict with familiar knowledge, so recall alone cannot solve the tasks. The benchmark contains two sandboxes, AlienCode (31 discovery targets, 70 tasks) and AlienLogic (24 discovery targets, 70 tasks). Each sandbox provides a flawed manual, task-specific environmental feedback, and a dedicated tool-call schema. Systems use these resources to explore the sandbox, then solve held-out tasks. We evaluate 10 AI systems and find that the strongest systems can acquire and apply unfamiliar rules, while performance varies substantially across trajectories and continued exploration can stall or reverse earlier gains. ExplorationBench represents a step towards AI systems that can acquire and apply genuinely new knowledge through exploration in unknown environments.
Text extracted automatically; images, tables and formatting may be missing. Original: https://arxiv.org/abs/2609.30199