Universal Textual Teaching for LLMs
Universal Textual Teaching distills teacher-student gaps into a reusable natural-language Primer without updating student weights.
Universal Textual Teaching is a parameter-update-free distillation method that converts observed teacher-student knowledge gaps into a reusable natural-language artifact called a Primer. Iterative roles have the student attempt tasks, a prompter turn feedback into teaching instructions, the teacher demonstrate, and a synthesizer consolidate lessons. On Omni-MATH-2, accuracy rose from 27.6% to 51.7%; on KernelBench, accuracy rose from 9.4% to 48.6% and Fast1 from 9% to 35%. Primers generalized to students not involved in synthesis and beat prompt-engineering and parameter-based distillation baselines.
- Distills gaps into a reusable text Primer without student weight updates.
- Omni-MATH-2 accuracy rose from 27.6% to 51.7%.
- KernelBench accuracy rose from 9.4% to 48.6%; Fast1 from 9% to 35%.
- Primers transferred to students not used during synthesis.
From the original article
1 min read at huggingface.co
Knowledge distillation (KD) transfers knowledge from stronger Teacher models to weaker Student models, but most methods require training the Student parameters, thereby binding the distilled knowledge to a specific architecture and checkpoint. This implicit representation is difficult to interpret or reuse across models and limits KD for API-only or costly-to-train models.
By Zhanyi Lu, Huan Wang · huggingface.co
A short excerpt; the article and its images belong to the publisher. Summary, key facts and assessment above are AI-written from the article and can be wrong.