NarrowSpec runs speculative drafts at half or quarter hidden width
Original title: NarrowSpec: Draft Hidden Width Need Not Match the Target in Speculative Decoding
On Qwen3-8B and GPT-OSS-20B, narrower DFlash and EAGLE-3 drafts sped decoding by up to 1.44x.
NarrowSpec shows that draft models which read a target LLM’s hidden states, including DFlash and EAGLE-3, can run their residual stream at half or a quarter of the target width while largely preserving acceptance. Tests used Qwen3-8B (NVFP4) and GPT-OSS-20B (MXFP4) in SGLang on a B200 and a DGX Spark (GB10). At batch size 1 and temperature 0, narrowing sped end-to-end decoding by 1.06–1.12x on B200 and 1.13–1.44x on GB10, and no narrow draft was slower than full width.
- For DFlash, the trained output adapter folds into one narrower projection.
- From batch 32 to the largest fitted batch, narrowing reaches up to 1.21x.
- Speedups held at larger batch sizes on both GPUs.
- Targets were Qwen3-8B (NVFP4) and GPT-OSS-20B (MXFP4).
- Products
- SGLang
- AI models
- Qwen3-8BGPT-OSS-20BDFlashEAGLE-3
From the original article
1 min read at huggingface.coSpeculative decoding is widely used to speed up generation in large language models (LLMs), with a small draft model proposing tokens that one forward pass of the target model verifies. Draft models that read the target's hidden states, such as DFlash and EAGLE-3, typically inherit the target's hidden width.
By Alagappan Valliappan, Ankit More, Dheevatsa Mudigere · huggingface.co
A short excerpt; the article and its images belong to the publisher. Summary, key facts and assessment above are AI-written from the article and can be wrong.