ProVe proposer-verifier decoding accelerates language models
Original title: ProVe: Proposer-Verifier Decoding
ProVe reaches 2.7x throughput using a 3B verifier at only 2.6 accuracy points below autoregressive baselines.
ProVe reverses speculative decoding by fine-tuning a large model as a proposer that predicts blocks of tokens. A verifier then checks the block in one forward pass. With a block size of 16, the proposer achieves an average prefix match of 6.6 tokens. ProVe-lossless delivers 2.5x throughput on 32B models without extra heads, while the small-verifier version provides a tunable speed-accuracy tradeoff across six benchmarks.
- Block proposals yield 6.6-token average AR-matching prefix.
- ProVe-lossless uses the proposer itself for lossless decoding.
- Small 3B verifier achieves 2.7x average throughput.
From the original article
1 min read at huggingface.co
Speculative decoding accelerates autoregressive (AR) language models by letting a small model draft tokens serially and a large model verify them in parallel. We reverse these roles. Starting from a standard AR checkpoint, we fine-tune the large model into a proposer that predicts a block of future tokens in a single forward pass, then use one additional forward pass to verify the proposed block.
By Ziqi Jin, Weiwen Xu, Jiaxi Li · huggingface.co
A short excerpt; the article and its images belong to the publisher. Summary, key facts and assessment above are AI-written from the article and can be wrong.