Do We Really Need KL Divergence for On-Policy Distillation of Large Language Models?
Researchers show on-policy LLM distillation works without KL divergence if updates move toward the teacher.
The paper argues KL divergence is unnecessary for on-policy distillation of large language models. Assigning +1 or -1 rewards based on whether the teacher probability exceeds the student's nearly matches reverse-KL on-policy distillation. Only tokens with large teacher-student disagreement need updates toward the teacher. Consensus Multi-Teacher On-Policy Distillation lets every sample be supervised by all teachers and outperforms single-teacher routing on math and code benchmarks.
- Direction toward the teacher, not full KL, suffices for OPD.
- A small set of high-disagreement tokens drives effective updates.
- Simple +1/-1 token rewards reproduce reverse-KL training behavior.
- C-MOPD beats single-teacher routing on math and code.
- Organizations
- LeapLabTHU
From the original article
1 min read at huggingface.co
Since the advent of knowledge distillation, KL divergence has been the standard loss in distillation. Recently, on-policy distillation (OPD) has emerged as an efficient post-training paradigm for LLMs. As a distillation method, OPD naturally inherits KL divergence as its standard loss.
By Wenze Lin, Jiyuan Long, Jiale Zhao · huggingface.co
A short excerpt; the article and its images belong to the publisher. Summary, key facts and assessment above are AI-written from the article and can be wrong.