On-Policy Distillation Teaches New Skills but Not New Knowledge
On-policy distillation transfers compositional reasoning skill but little new factual knowledge under reverse KL.
Stories per day · 30 days
2 stories, newest first · one card per story
Stories tagged #reverse-kl
Rethinking Cross-Tokenizer On-Policy Distillation: From Alignment Coverage to Supervision Reliability
Cross-tokenizer on-policy distillation works better with strict token alignment than broader, less reliable supervision.