The Missing Primitive: Diagnosing and Repairing Mathematical Reasoning in Large Language Models
A new benchmark and self-distillation method diagnose and repair weak mathematical reasoning in LLMs.
The paper defines Mathematical Primitives and a benchmark that scores LLM math reasoning on Discovery, Generation, Digestion, and Execution. Diagnosis finds solution accuracy hides distinct capability profiles, primitives unlock latent execution, and Discovery is the main bottleneck. Discovery-limited failures are especially repairable. A primitive-privileged self-distillation method improves reasoning over baselines across model scales and hard benchmarks.
- Benchmark scores Discovery, Generation, Digestion, and Execution.
- Accuracy masks distinct capability profiles; Discovery is the bottleneck.
- Primitives unlock substantial latent execution capacity.
- Self-distillation repairs discovery-limited failures across model scales.
From the original article
1 min read at arxiv.orgWhile Large Language Models (LLMs) have demonstrated striking capabilities on frontier mathematical problems, it remains unclear whether they possess the structural mathematical understanding underlying their solutions. In this paper, we take a first step toward systematically studying mathematical understanding in LLMs, from diagnosing its distinct capabilities to leveraging these findings to improve post-training.
By Shuo Xing · arxiv.org
A short excerpt; the article and its images belong to the publisher. Summary, key facts and assessment above are AI-written from the article and can be wrong.