The Missing Primitive: Diagnosing and Repairing Mathematical Reasoning in Large Language Models
A primitive-based diagnosis finds Discovery is the main bottleneck in LLM mathematical reasoning and shows it can be repaired.
The paper defines Mathematical Primitives to probe structural understanding and evaluates LLM reasoning across Discovery, Generation, Digestion, and Execution. Diagnosis shows accuracy hides distinct capability profiles, primitives unlock latent execution capacity, and Discovery is the main bottleneck. A primitive-privileged self-distillation framework transfers primitive-guided reasoning and improves results over baselines across model scales and hard benchmarks.
- Four dimensions: Discovery, Generation, Digestion, and Execution.
- Solution accuracy masks distinct capability profiles across primitives.
- Discovery is the dominant bottleneck and is especially repairable.
- Primitive-privileged self-distillation improves reasoning across scales and benchmarks.
From the original article
1 min read at huggingface.co
While Large Language Models (LLMs) have demonstrated striking capabilities on frontier mathematical problems, it remains unclear whether they possess the structural mathematical understanding underlying their solutions. In this paper, we take a first step toward systematically studying mathematical understanding in LLMs, from diagnosing its distinct capabilities to leveraging these findings to improve post-training.
By Shuo Xing, Zilin Dai, Chengyuan Qian · huggingface.co
A short excerpt; the article and its images belong to the publisher. Summary, key facts and assessment above are AI-written from the article and can be wrong.