To Copy or Not to Copy: Controlling Speculative Decoding via Intrinsic Model Signals
SwitchSD uses internal LLM signals to adaptively switch between neural drafting and context-based copying in speculative decoding, boosting throughput up to 15%.
Researchers introduce SwitchSD, an adaptive speculative decoding framework that treats copying as a latent control signal of the target LLM. Lightweight probes trained on internal representations identify genuine copy-intent with AUC above 0.99, allowing dynamic switching between neural drafting (EAGLE/EAGLE3) and context-based copying. Experiments across the Llama and Qwen families show throughput gains up to 15% over state-of-the-art baselines like EAGLE3 by eliminating false-positive copy triggers from accidental repetitions.
- Treats copying as a latent control signal detected from model internals
- Probes identify true copy-intent with AUC greater than 0.99
- Fixes false-positive triggers caused by surface n-gram repetition
- Up to 15% throughput gain over EAGLE3 on Llama and Qwen models
- AI models
- SwitchSD
From the original article
1 min read at arxiv.orgSpeculative Decoding (SD) has significantly accelerated Large Language Model (LLM) inference, yet existing approaches face a fundamental tradeoff between two drafting strategies: neural drafting and context-based copying. Neural drafts (e. g.
By Roy Eisenstadt · arxiv.org
A short excerpt; the article and its images belong to the publisher. Summary, key facts and assessment above are AI-written from the article and can be wrong.