SpecOPD branch-parallel distillation accelerates speculative decoding
Original title: SpecOPD: Branch-Parallel On-Policy Distillation for Speculative Decoding
SpecOPD parallelizes on-policy distillation across branches for up to 41.45% higher mean acceptance length and 3.56times generation speedup.
SpecOPD uses branch-parallel on-policy distillation to avoid sequential draft-and-verify dependencies in speculative decoding. It packs branches into proposal trees sharing computation and achieves higher throughput across model sizes and architectures. Results span dialogue coding mathematics and agentic tasks.
- Up to 41.45% higher mean acceptance length
- 16.09times supervised-token throughput over sequential OPD
- 3.56times generation speedup over autoregressive decoding
- Spans 8B 27B and 125B targets
- Uses E2E-TV surrogate for acceptance length
- AI models
- SpecOPD
From the original article
1 min read at huggingface.co
Speculative decoding (SD) accelerates large language model inference by reducing the sequential cost of token-by-token generation. Advanced SD methods reduce proposal cost with parallel drafters that predict entire blocks in one forward pass, but overall speedup still depends on acceptance length.
By Tianyu Liu, Jun Zhang, Wenjie Huang · huggingface.co
A short excerpt; the article and its images belong to the publisher. Summary, key facts and assessment above are AI-written from the article and can be wrong.