RCSD raises speculative-decoding acceptance length 35% over HASS
Original title: RCSD: Reinforcement and Continual Learning for Domain-Specific Speculative Decoding
Draft models trained on accepted and rejected tokens report 1.35 times faster inference without a quality loss.
RCSD trains speculative-decoding draft models with reinforcement and continual learning to maximize acceptance length, instead of soft-label distillation of the full target distribution. Accepted and rejected sequences from speculative inference become implicit preference pairs, with masked loss on tokens already aligned with the target. The paper reports a 35% higher average acceptance length than HASS and 1.35x faster end-to-end inference on practical workloads.
- Soft-label distillation can copy high-entropy target tokens and raise early rejection.
- Masked loss avoids penalizing tokens that already match the target.
- The authors say generation quality is not compromised.
From the original article
1 min read at arxiv.orgSpeculative decoding accelerates large language model inference by employing a compact draft model to propose candidate tokens that are verified in parallel by a target model. Conventional training relies on soft-label distillation, which forces the draft model to mimic the target's full output distribution, including high-entropy regions that degrade acceptance rates and increase early rejection risk.
By Mingkang Liu · arxiv.org
A short excerpt; the article and its images belong to the publisher. Summary, key facts and assessment above are AI-written from the article and can be wrong.