Recovering Reliable Supervision via Critique-Guided On-Policy Distillation
CriticOPD uses teacher critiques to restore recoverability in on-policy distillation, lifting two Qwen3 students on math benchmarks.
On-policy distillation trains a student on its own rollouts with token-level teacher supervision, assuming the teacher can reach a correct answer from any student prefix. The authors show this recoverability fails on incorrect rollouts because the teacher follows student errors even when it recognizes them. CriticOPD has the teacher critique the error and the student rewrite the solution, while training uses the revision without the critique. On two Qwen3 students and six mathematical reasoning benchmarks, CriticOPD beat standard on-policy distillation, with the largest gains on competition problems.
- On-policy distillation assumes a teacher can recover from any student prefix.
- Teachers follow recognized student errors instead of correcting incorrect rollouts.
- CriticOPD critiques the error and trains on the student's critique-free revision.
- Largest gains were on competition problems for two Qwen3 students.
- AI models
- Qwen3
From the original article
1 min read at arxiv.orgOn-policy distillation (OPD) trains a student model on its own rollouts with token-level supervision from a teacher, assuming that the teacher can reach a correct answer from any prefix the student writes. We call this property the recoverability of the teacher and show it fails on incorrect rollouts: the teacher follows student errors, although it recognizes them.
By Kaixiang Zhao · arxiv.org
A short excerpt; the article and its images belong to the publisher. Summary, key facts and assessment above are AI-written from the article and can be wrong.