Soft-MTP drops a noisy residual in multi-token prediction training
Original title: Multi-Token Prediction Done Right
From 167M to 1.2B parameters, Soft-MTP lowers next-token perplexity and speeds speculative decoding without new parameters.
The paper says standard multi-token prediction mixes a useful KL self-distillation signal with a low signal-to-noise residual that grows with scale. Soft-MTP recovers only the KL gradient by routing existing next-token and multi-token cross-entropy gradients, adding no parameters and no measurable overhead. From 167M to 1.2B parameters it beats standard MTP on perplexity, speculative decoding, and zero-shot accuracy, and improves math and code after fine-tuning at 1.2B.
- The method supports DeepSeek-style and Meta-style multi-token prediction.
- It reuses fused cross-entropy kernels and avoids vocabulary-sized logit matrices.
- Lower multi-token perplexity appears from 689M parameters onward.
- Code is published at github.com/JaydenTeoh/Soft-MTP.
From the original article
1 min read at huggingface.co
Multi-Token Prediction (MTP) is now a common ingredient in language-model pre-training, as it improves downstream performance and enables faster inference through speculative decoding. We identify a flaw in standard MTP training, in which a residual component of the MTP gradient interferes with both next-token and multi-token learning, increasingly so with scale.
By Jayden Teoh · huggingface.co
A short excerpt; the article and its images belong to the publisher. Summary, key facts and assessment above are AI-written from the article and can be wrong.