Reward-free repair targets myopic scoring in agentic distillation
Original title: Capable Teacher, Myopic Scorer: Reward-Free Teacher Repair for Agentic On-Policy Distillation
R2S beat three agentic on-policy methods in all 12 teacher-student-environment cells, by 12.8 success points on average.
The paper argues that agentic on-policy distillation fails partly because next-token teacher scores miss how an episode ends, not only because student errors leave the teacher’s reliable region. Rollout-to-Scorer Distillation mines high-disagreement states, keeps teacher takeovers that pass a reward-free check, and fine-tunes the teacher’s scores before freezing it as the sole scorer. Repair and training use no task reward.
Unconfirmed: The 12.8-point average gain is the authors’ reported result, not an independent replication.
- Authors name the mismatch between teacher actions and token scores the rollout-scorer gap.
- Checks used to keep takeovers include consensus or an LLM judge.
- The repaired teacher scores every assistant token from the divergence turn without the takeover in context.
- Evaluation used three environments, two teachers and four students.
From the original article
1 min read at arxiv.orgAgentic on-policy distillation (OPD) trains a student agent on its own trajectories while a teacher scores every token.
By Shu Zhao · arxiv.org
A short excerpt; the article and its images belong to the publisher. Summary, key facts and assessment above are AI-written from the article and can be wrong.