Steered-teacher self-distillation cuts reference hallucinations in thinking models
Original title: Steering the Teacher: Improving On-Policy Self-Distillation for Thinking Models
Activation steering is applied only to the teacher; the student is not steered, and AIME and HMMT accuracy rises over vanilla OPSD.
On-policy self-distillation can make thinking models cite unavailable references and skip verification when the teacher sees a reference. Steered-Teacher OPSD extracts two directions from model-generated reasoning and applies them to the teacher's hidden states while computing distillation targets. The paper reports higher accuracy than vanilla OPSD and other interventions on AIME and HMMT, with reconsideration and reference-hallucination rates restored to base-model levels.
Unconfirmed: Accuracy and rate claims are the authors' reported experimental results, not independently verified.
- Access to a reference can induce reference hallucinations and reconsideration suppression.
- Two steering directions suppress reference-invoking behavior and encourage reconsideration.
- Marker-family calibration is used when encouraging reconsideration.
- The student learns from steered teacher distributions without itself being steered.
From the original article
1 min read at arxiv.orgOn-policy self-distillation (OPSD) trains a student on its own reasoning trajectories using token-level supervision from a reference-conditioned teacher initialized from the same model.
By Tiejin Chen · arxiv.org
A short excerpt; the article and its images belong to the publisher. Summary, key facts and assessment above are AI-written from the article and can be wrong.