Speculative diffusion paper maps joint speed-quality tradeoffs
Original title: Speculative Diffusion is a Tale of Two Tradeoffs
Authors report that jointly varying diffusion effort and acceptance leniency beats single-step diffusion drafting on the speed-quality frontier.
The paper studies speculative decoding when diffusion language models are used as draft models. Classic speculative decoding stays lossless relative to the verifier, while diffusion models can trade quality for speed by using less diffusion effort. Speculative decoding can also be made lossy with more lenient acceptance. Jointly varying those parameters extends the Pareto frontier beyond single-step diffusion drafting, the authors report.
Unconfirmed: The extended Pareto frontier is the authors' experimental claim, not an independent replication.
- Classic speculative decoding is lossless with respect to the verifier distribution.
- Diffusion models generate tokens in parallel and can reduce diffusion effort for speed.
- More lenient acceptance criteria yield faster but lossy samples.
- Joint sweeps of both parameters extend the frontier past single-step diffusion drafting.
From the original article
1 min read at huggingface.co
Speculative decoding is a technique that accelerates the inference speed of autoregressive large language models (LLMs) by sampling from a faster, lower quality draft model. Traditionally, a fundamental characteristic of speculative decoding is that it is lossless with respect to the verifier distribution, guaranteeing speed without sacrificing quality.
By Aaron Zorel, Guy Van den Broeck, Aditya Grover · huggingface.co
A short excerpt; the article and its images belong to the publisher. Summary, key facts and assessment above are AI-written from the article and can be wrong.