Do We Really Need KL Divergence for On-Policy Distillation of Large Language Models?
Researchers show on-policy LLM distillation works without KL divergence if updates move toward the teacher.
1 stories, newest first · one card per story
Researchers show on-policy LLM distillation works without KL divergence if updates move toward the teacher.
Today 1.6K readers · 2.5K views · 779 reads · all time 189K views