Skip to content
NewA calmer, faster ZeroHour

NarrowSpec runs speculative drafts at half or quarter hidden width

Original title: NarrowSpec: Draft Hidden Width Need Not Match the Target in Speculative Decoding

Hugging Face daily papersby Alagappan Valliappan, Ankit More, Dheevatsa Mudigere ()1 min read1
AI summary · x-ai/grok-4.7Summaries by Vani AI

On Qwen3-8B and GPT-OSS-20B, narrower DFlash and EAGLE-3 drafts sped decoding by up to 1.44x.

NarrowSpec shows that draft models which read a target LLM’s hidden states, including DFlash and EAGLE-3, can run their residual stream at half or a quarter of the target width while largely preserving acceptance. Tests used Qwen3-8B (NVFP4) and GPT-OSS-20B (MXFP4) in SGLang on a B200 and a DGX Spark (GB10). At batch size 1 and temperature 0, narrowing sped end-to-end decoding by 1.06–1.12x on B200 and 1.13–1.44x on GB10, and no narrow draft was slower than full width.

Key facts
  • For DFlash, the trained output adapter folds into one narrower projection.
  • From batch 32 to the largest fitted batch, narrowing reaches up to 1.21x.
  • Speedups held at larger batch sizes on both GPUs.
  • Targets were Qwen3-8B (NVFP4) and GPT-OSS-20B (MXFP4).
Explain this AI news with Vani AIWhat was announced, what is new and why it matters, in a new chat

From the original article

1 min read at huggingface.co

Speculative decoding is widely used to speed up generation in large language models (LLMs), with a small draft model proposing tokens that one forward pass of the target model verifies. Draft models that read the target's hidden states, such as DFlash and EAGLE-3, typically inherit the target's hidden width.

Hugging Face daily papers

By Alagappan Valliappan, Ankit More, Dheevatsa Mudigere · huggingface.co

Read the full article at Hugging Face daily papers

A short excerpt; the article and its images belong to the publisher. Summary, key facts and assessment above are AI-written from the article and can be wrong.