Understanding On-Policy Distillation: A Mechanistic Interpretability Perspective via Sparse Crosscoders
Sparse crosscoders show on-policy distillation reweights features LLMs already share rather than creating new ones.
1 stories, newest first · one card per story
Sparse crosscoders show on-policy distillation reweights features LLMs already share rather than creating new ones.
Today 1.6K readers · 2.5K views · 779 reads · all time 189K views