Order-Invariant Answers, Order-Sensitive Representations in Mathematical Reasoning
Across 16 models from 1B to 8B, higher math accuracy tracks more distinct internal encodings of equivalent rule orders.
The paper tests whether language models preserve internal representations when mathematical rules are reordered without changing the correct answer, using synthetic multi-step function-composition problems. Across 16 models from 1B to 8B parameters, models that solve reordered problems more accurately also encode different rule orderings more distinctly. Layer-averaged permutation signal-to-noise ratio is positively rank-correlated with accuracy in every setting, with Spearman correlations reaching 0.86. The authors argue that answer invariance and representation invariance should be measured separately.
- Tasks reorder equivalent math rules while keeping the correct answer fixed.
- The study evaluates 16 language models from 1B to 8B parameters.
- Permutation SNR correlates with accuracy, reaching Spearman 0.86.
- Correct answers can coexist with order-sensitive internal representations.
Full article156 words · extracted from arxiv.org · click to collapse
Reordering a set of mathematical rules without changing its meaning should preserve the correct answer, but must a model's internal representations stay invariant too? We investigate this question using synthetic multi-step function-composition problems, each presented under multiple rule orderings with the same correct answer. We measure accuracy and permutation signal-to-noise ratio (SNR), which quantifies how distinctly ordering patterns are represented relative to variation across problem instances. Across 16 language models ranging from 1B to 8B parameters, we find a pattern: models that solve reordered problems more accurately represent different rule orderings more distinctly. Layer-averaged permutation SNR is positively rank-correlated with accuracy in every synthetic setting we evaluate, with Spearman correlations reaching 0.86. These findings highlight a distinction between answer invariance and representation invariance: successful mathematical rule composition can accompany distinct internal representations between equivalent rule orderings. This motivates distinguishing answer invariance from representation invariance, and offers a representational perspective on mathematical reasoning beyond answer accuracy alone.
Text extracted automatically; images, tables and formatting may be missing. Original: https://arxiv.org/abs/2609.28442