On-Policy Distillation Teaches New Skills but Not New Knowledge
On-policy distillation transfers compositional reasoning skill but little new factual knowledge under reverse KL.
Researchers separate factual knowledge from compositional skill in on-policy distillation using a controlled synthetic framework. Across four models from three families, reverse-KL distillation transfers skill on unseen reasoning structures but minimal new facts. Switching to forward KL restores factual transfer, while student rollouts improve multi-step execution. Factual QA and competition-math experiments show reasoning gains without expanding parametric memory.
- Reverse-KL on-policy distillation transfers compositional skill, not new facts
- Forward KL restores factual knowledge transfer
- Student rollouts specifically improve multi-step reasoning execution
- The same asymmetry appears on factual QA and competition mathematics
From the original article
1 min read at huggingface.co
On-policy distillation (OPD) strengthens language-model reasoning, yet whether students acquire new factual knowledge or compositional skill for multi-step reasoning remains unknown. We separate these capabilities using a controlled synthetic framework that measures the student's initial capabilities and independently controls the teacher's additional facts, compositional skill, or both.
By Yixuan Tang, Yi Yang · huggingface.co
A short excerpt; the article and its images belong to the publisher. Summary, key facts and assessment above are AI-written from the article and can be wrong.