Teacher-Evolution Self-Distillation improves on-policy model training
Original title: On-Policy Self-Distillation with Teacher Evolution
TESD evolves self-teachers with task rewards to provide reliable guidance and down-weights unreliable samples.
On-Policy Self-Distillation often fails to deliver continuous gains because self-teachers generate unreliable guidance. TESD fixes this by optimizing the teacher with environmental feedback on correctness and context independence. Adaptive student optimization then reduces the impact of low-reliability samples. Experiments confirm TESD outperforms prior token-level and sequence-level approaches across datasets and model scales.
- Teacher-oriented optimization enhances self-teaching capability.
- Adaptive optimization down-weights unreliable guidance samples.
- Consistent outperformance on multiple datasets and scales.
From the original article
1 min read at huggingface.co
On-Policy Self-Distillation (OPSD), which aligns a student model's token-level distribution with that of its self-teacher prompted by privileged context, has emerged as a promising direction to mitigate reward sparsity in sequence-level post-training paradigms. However, OPSD often fails to achieve continuous performance gains during training.
By Shufan Shen, Zhongni Hou, Chao Bi · huggingface.co
A short excerpt; the article and its images belong to the publisher. Summary, key facts and assessment above are AI-written from the article and can be wrong.