Strong-to-weak OPD achieves 50x compute efficiency versus GRPO when using own checkpoint
Original title: Unpacking the Compute Efficiency of Strong-to-Weak OPD against GRPO
On-policy distillation with the student's own GRPO checkpoint reaches target accuracy with 50 times fewer FLOPs than GRPO training.
The paper compares on-policy distillation (OPD) with Group Relative Policy Optimization (GRPO) across task difficulty and model sizes. OPD with an external post-trained teacher loses to GRPO in accuracy per FLOP but with the student's own GRPO checkpoint OPD recovers accuracy on math with 50x fewer distillation FLOPs. Multi-teacher OPD consolidates per-domain experts with 42% fewer total FLOPs than joint GRPO.
- OPD with own GRPO checkpoint achieves 50x fewer distillation FLOPs
- Multi-teacher OPD reaches joint GRPO performance on 2-domain mix with 42% fewer total FLOPs
- OPD suffers response-length inflation with external teachers
- Separate GRPO checkpoints hold different capabilities across domains
From the original article
1 min read at arxiv.orgOn-policy distillation (OPD) trains a student on its own rollouts under a frozen teacher's per-token supervision. Does this dense signal reduce the training FLOPs needed to reach a target accuracy compared with reinforcement learning?
By Zeyu Leo Liu · arxiv.org
A short excerpt; the article and its images belong to the publisher. Summary, key facts and assessment above are AI-written from the article and can be wrong.