Branching On-Policy Distillation
BranchOPD branches uncertain on-policy rollouts, beating vanilla OPD by 18.55 math and 13.8 ALFWorld points.
BranchOPD extends on-policy distillation by selecting decision points with student uncertainty and teacher-student disagreement, resampling continuations from shared prefixes, and applying token-level teacher supervision to main trajectories and branches. Shared-prefix reuse reduces student-generated token cost by about 19% on ALFWorld and 31% on WebShop versus seven independent trajectories. Protocol-boundary repair adds targeted supervision to limit repetition. BranchOPD beats Vanilla OPD and Multi-Rollout OPD (n=7) on all reported benchmarks, including +18.55 percentage points avg@16 math accuracy and +13.8 points mean ALFWorld ID/OOD success.
- BranchOPD branches at high student uncertainty or teacher-student disagreement and supervises both paths.
- Shared-prefix reuse cut student token cost about 19% on ALFWorld and 31% on WebShop versus seven rollouts.
- Protocol-boundary repair targets repetition and wasted generation under protocol constraints.
- Gains over Vanilla OPD were 18.55 points avg@16 math accuracy and 13.8 points mean ALFWorld success.
- Uncertain-decision branching beat random selection and turn-initial branching in ablations.
From the original article
1 min read at arxiv.orgOn-policy distillation (OPD) provides teacher supervision on student-generated trajectories, but a single trajectory realizes only one continuation at each decision, limiting supervision coverage under a finite sampling budget.
By Dongyang Chen · arxiv.org
A short excerpt; the article and its images belong to the publisher. Summary, key facts and assessment above are AI-written from the article and can be wrong.