Smaller Models, Better Rejects: Preference Distillation Scaling
Smaller frozen models produce more useful preference-distillation rejects than a student's own failures on code and math.
The paper challenges the assumption that a student's own failures are the best rejected responses for preference distillation. Across students from 7B to 72B, smaller frozen models produced stronger training rejects at lower inference cost on code generation and mathematical reasoning, both before and after sequence-level distillation. A finite-horizon Direct Preference Optimization bound motivates mixing smaller-model rejects, preserving task structure, and selecting lower-likelihood candidates under the reference policy.
- Smaller frozen models beat self-generated rejects for 7B-72B students
- Gains hold before and after sequence-level distillation
- More small-model rejects in the mix improve performance
- Lower reference-policy likelihood candidates transfer better
From the original article
1 min read at huggingface.co
Preference distillation typically treats a teacher response as preferred and the student's own response as rejected. This assumes that self-generated failures are the most informative negatives and that rejects must come from a model at least as large as the student, making generation costly at scale.
By Rui Cai, Wenhui Zhu, Xiwen Chen · huggingface.co
A short excerpt; the article and its images belong to the publisher. Summary, key facts and assessment above are AI-written from the article and can be wrong.