Lossy speculative decoding inflates completion length
Original title: Acceptance Is Not Speed: Length Inflation in Training-Free Lossy Speculative Decoding
At the most relaxed settings tested, models accepted 1.34× tokens per round but wrote 1.54× longer completions.
A paper argues that accepted draft tokens per round is a poor speed metric for training-free lossy speculative decoding. Relaxed verification can accept tokens lossless checks would reject and lengthen completions, mainly by extending reasoning traces. Across twelve target-drafter pairs, the most relaxed setting accepted 1.34× as many tokens per round but produced 1.54× longer completions. Rules that cut verifier rounds typically cut accuracy substantially.
- Evaluation measured length, verifier rounds, wall-clock time, and accuracy.
- Extra accepted tokens often had low target-model probability.
- Some failures came from reasoning that exhausted the answer budget.
From the original article
1 min read at arxiv.orgSpeculative decoding typically characterizes speedup by draft tokens accepted per verification round. Lossless verification preserves the target model's output distribution and thus expected completion length, making this metric a meaningful efficiency proxy. Relaxed (lossy) verification accepts tokens that lossless verification would reject, potentially altering that distribution.
By Chiatzen Wang · arxiv.org
A short excerpt; the article and its images belong to the publisher. Summary, key facts and assessment above are AI-written from the article and can be wrong.