Making LLMs Say What They Think: Measuring and Improving CoT-Interpretability Alignment
A new metric shows LLMs' chain-of-thought often mismatches internal reasoning, and post-training can improve that alignment.
The paper proposes CoT-Interpretability Alignment (CIA), a metric comparing an LLM's written chain-of-thought with internal reasoning strategies detected by interpretability tools. On two-hop question answering, hint intervention, and integer multiplication across three models, CIA ranged from 44.8% to 75.9%. Post-training that rewards both task accuracy and parametric faithfulness substantially improved CoT faithfulness while maintaining or improving accuracy. Code and data are released for auditing explicit reasoning.
- CIA compares written CoT with internal strategies found by interpretability tools.
- Scores were 44.8–75.9% across three tasks and three LLMs.
- Post-training rewarded task accuracy and parametric faithfulness together.
- Faithfulness rose while accuracy held or improved; generalization was analyzed.
From the original article
1 min read at huggingface.co
Chain-of-thought (CoT) traces often serve as a proxy for how Large Language Models (LLMs) arrive at their answers. However, growing evidence shows that models' CoT often fails to reflect their internal computations and can be changed without affecting their final answers. In this work, we measure and improve the alignment between the reasoning described in an LLM's CoT and what it computes internally.
By Yihuai Hong, Shauli Ravfogel, Chen Zhao · huggingface.co
A short excerpt; the article and its images belong to the publisher. Summary, key facts and assessment above are AI-written from the article and can be wrong.