OC-OPD improves on-policy distillation via observation-coordinated supervision
Original title: OC-OPD: Observation-coordinated Supervision for On-Policy Distillation
OC-OPD uses reverse and forward KL divergences with a four-observation matrix to coordinate gradients, achieving strong gains in mathematical reasoning and OOD transfer.
Reverse and forward KL divergences complement each other in on-policy distillation. OC-OPD forms a four-observation matrix from student and teacher at student-generated prefixes. It equates Fisher inverse-metric coupling of logit gradients with observation contrasts and uses closed-form bargaining to scale updates. Across non-thinking, thinking, and base-model initialization settings, OC-OPD shows gains in mathematical reasoning and transfers to scientific QA and code generation. Code is publicly available.
- Diagonal self-observations measure uncertainty
- Cross-observations recover reverse and forward KL
- Exact identity links gradients to observation contrasts
- Strong gains in mathematical reasoning
- Effective transfer to out-of-distribution scientific QA and code
From the original article
1 min read at arxiv.orgReverse and forward Kullback--Leibler (KL) divergences provide complementary supervision in on-policy distillation: reverse KL refines the student's active token rankings, and forward KL allocates mass to teacher-supported candidates. At student-generated prefixes, we form a four-observation matrix by treating the student and teacher as both candidate sources and evaluators.
By Ruoran Xu · arxiv.org
A short excerpt; the article and its images belong to the publisher. Summary, key facts and assessment above are AI-written from the article and can be wrong.