Muon-RT Reuses Teacher Trajectory for Momentum-Free Muon
Original title: MUON-RT Reusing the Teacher's Trajectory for Momentum-Free Muon for LLM Self-Distillation
Muon-RT eliminates momentum buffers by reusing the teacher-student displacement in self-distillation.
Self-distillation methods for large language models use an exponential moving average teacher that retains update history. Muon-RT reuses this displacement to remove separate momentum buffers for hidden weight matrices. It improves accuracy over baselines with reduced persistent memory.
From the original article
1 min read at huggingface.co
Self-distillation methods for large language models with an exponential moving average (EMA) teacher already retain a history of student updates. Muon nevertheless stores a separate momentum buffer, increasing persistent training memory. We propose Muon with Reused Teacher (Muon-RT), which reuses the teacher-student displacement to eliminate momentum buffers for hidden weight matrices.
By Aryan Das, Vinay Kumar Verma, Aakansha Mishra · huggingface.co
A short excerpt; the article and its images belong to the publisher. Summary, key facts and assessment above are AI-written from the article and can be wrong.