ExplorationBench: Measuring AI Systems' Exploration in Verifiable Alien Worlds
ExplorationBench evaluates AI systems' scientific exploration in verifiable 'Alien Worlds' sandboxes where recall cannot solve tasks; top systems still stall or reverse gains.
ExplorationBench measures scientific exploration ability using executable 'Alien Worlds' whose rules conflict with pre-training knowledge, so tasks cannot be solved by recall alone. It ships two sandboxes, AlienCode (31 discovery targets, 70 tasks) and AlienLogic (24 discovery targets, 70 tasks), each with a flawed manual, environmental feedback, and a tool-call schema. Across 10 AI systems evaluated, the strongest can acquire and apply unfamiliar rules, but performance varies substantially across trajectories and continued exploration can stall or reverse earlier gains.
- Two executable sandboxes (AlienCode, AlienLogic) with 140 tasks over 55 discovery targets
- Rules conflict with pre-training knowledge, preventing recall-based solutions
- 10 systems evaluated; strongest learn unfamiliar rules but gains can stall or reverse
- Organizations
- ExplorationBench
From the original article
1 min read at huggingface.co
Scientific discovery begins where known problems end. There, AI systems must engage in exploration: framing hypotheses, designing experiments, and iterating on the results. However, evaluating this ability is difficult: (1) how to verify whether a genuinely new hypothesis holds, and (2) how to determine whether a system has discovered it through exploration or merely recalled related knowledge from pre-training data.
By Ming Zhang, Zhenghao Xiang, Peizhong Gao · huggingface.co
A short excerpt; the article and its images belong to the publisher. Summary, key facts and assessment above are AI-written from the article and can be wrong.