1% of Tokens Can Be Enough: On Gradient Estimation in On-Policy Distillation
An information-efficiency ratio lets sparse on-policy distillation match full supervision at 0.1%–1% of tokens.
1 stories, newest first · one card per story
An information-efficiency ratio lets sparse on-policy distillation match full supervision at 0.1%–1% of tokens.
Today 1.6K readers · 2.4K views · 774 reads · all time 189K views