What Does Privileged Information Add to On-Policy Self-Distillation?
Study of on-policy self-distillation on 5,319 AMPLE-Math problems finds reference-free distillation drives most gains, with modest extra benefit from privileged references.
1 stories, newest first · one card per story
Study of on-policy self-distillation on 5,319 AMPLE-Math problems finds reference-free distillation drives most gains, with modest extra benefit from privileged references.
Today 1.6K readers · 2.5K views · 780 reads · all time 189K views