Order-Invariant Answers, Order-Sensitive Representations in Mathematical Reasoning
Across 16 models from 1B to 8B, higher math accuracy tracks more distinct internal encodings of equivalent rule orders.
The paper tests whether language models preserve internal representations when mathematical rules are reordered without changing the correct answer, using synthetic multi-step function-composition problems. Across 16 models from 1B to 8B parameters, models that solve reordered problems more accurately also encode different rule orderings more distinctly. Layer-averaged permutation signal-to-noise ratio is positively rank-correlated with accuracy in every setting, with Spearman correlations reaching 0.86. The authors argue that answer invariance and representation invariance should be measured separately.
- Tasks reorder equivalent math rules while keeping the correct answer fixed.
- The study evaluates 16 language models from 1B to 8B parameters.
- Permutation SNR correlates with accuracy, reaching Spearman 0.86.
- Correct answers can coexist with order-sensitive internal representations.
From the original article
1 min read at arxiv.orgReordering a set of mathematical rules without changing its meaning should preserve the correct answer, but must a model's internal representations stay invariant too? We investigate this question using synthetic multi-step function-composition problems, each presented under multiple rule orderings with the same correct answer.
By Zhixu Silvia Tao · arxiv.org
A short excerpt; the article and its images belong to the publisher. Summary, key facts and assessment above are AI-written from the article and can be wrong.