Thought without systematicity? Evaluating reasoning models on rule induction tasks
Study finds reasoning models often fail on structurally equivalent variants of tasks they solve, suggesting their reasoning lacks systematicity.
The paper extends rule induction tasks from cognitive science using task isomorphisms such as recombination and substitution to test systematicity in reasoning models. Despite solving tasks correctly, models frequently fail on structurally equivalent variants of the same task. The authors conclude many model behaviors lack systematicity, making it difficult to establish cognitive abilities beyond the specific evaluation contexts.
- Extends cognitive-science rule induction tasks via isomorphisms like recombination and substitution
- Models that solve a task often fail structurally equivalent variants
- Questions robustness of claimed reasoning abilities in current models
Full article140 words · extracted from huggingface.co · click to collapse
A central tenet of human cognition is systematicity, the principle that understanding one concept is inherently tied to understanding close variations of that concept. Do reasoning models robustly exhibit such systematicity? If so, we would expect consistent performance on structurally equivalent variants of the same task. Here, we extend established rule induction tasks from cognitive science to assess the systematicity of thought in current reasoning models. Each task family has compositional structure that we use to create structurally equivalent task variations through task isomorphisms such as recombination and substitution. We find that despite being able to correctly solve a task, models often fail on structurally equivalent variants of the same task. These findings suggest that many model behaviors lack systematicity, rendering it difficult to robustly establish the cognitive abilities of reasoning models beyond the particular contexts they were evaluated in.
Text extracted automatically; images, tables and formatting may be missing. Original: https://huggingface.co/papers/2609.13948