A Good Self-Teacher Meets the Student Where They Are: Joint On-Policy Learning and Teaching
JOLT jointly trains a privileged self-teacher and student so on-policy distillation matches the student's ability.
The paper studies self-distillation where one policy, conditioned on privileged information, teaches its unprivileged self. It shows privileged shortcuts can make a stronger teacher produce updates that hurt the student, and derives when a local distillation update is a positive multiple of the student's reward gradient. JOLT trains the teacher with outcome rewards plus token-level KL regularization toward the student, and the student with dense on-policy distillation. Experiments on math, coding, tool use, and terminal use report better training efficiency and performance, with further gains when student rewards are added.
- Privileged teachers can use shortcuts the student cannot follow.
- A KL-to-student term keeps teacher guidance capability-matched.
- JOLT jointly trains privileged teacher and unprivileged student roles.
- Gains reported on math, coding, tool use, and terminal tasks.
From the original article
1 min read at arxiv.orgReinforcement Learning (RL) from outcome rewards suffers from sparse supervision, particularly on difficult, long-horizon tasks where successful trajectories are rare and costly to generate. On-Policy Distillation (OPD) offers an attractive alternative by providing dense token-level supervision from a stronger teacher along the student's own generations.
By Randy Ardywibowo · arxiv.org
A short excerpt; the article and its images belong to the publisher. Summary, key facts and assessment above are AI-written from the article and can be wrong.