Adaptive multi-token prediction selects draft horizons online
Original title: Adaptive Multi-Token Prediction: How far should we go?
On Llama and Qwen pairings, it was up to 4.1x faster than standard decoding and beat a fixed window in 8 of 10 settings.
Researchers present Adaptive Multi-Token Prediction, a controller that chooses how many tokens to draft before one verification pass. Because all heads fire together, the horizon must be fixed in advance; the controller uses acceptance feedback and online gradient descent without retraining the verifier. Across five pairings built on Llama-3.1-8B-Instruct, Medusa heads, Meta MTP models and Qwen3-4B-Inst-2507-MTP, tests on AIME 2025 and GSM8K showed throughput gains of up to 10% versus a fixed window.
- AMTP matched or outperformed the fixed window in 8 of 10 settings.
- Acceptance survival estimates whether a draft prefix is fully accepted.
- Online gradient descent yields sublinear regret against the best horizon.
- The exact verifier is left unchanged and no retraining is required.
- Organizations
- Meta
From the original article
1 min read at arxiv.orgMulti-token prediction (MTP) can accelerate language-model decoding by drafting several tokens and verifying them in a single target-model pass. However, the best speculation horizon changes with context: longer drafts offer greater amortization when reliable, but can waste compute on suffixes discarded after a rejection.
By Emile Anand · arxiv.org
A short excerpt; the article and its images belong to the publisher. Summary, key facts and assessment above are AI-written from the article and can be wrong.