To Copy or Not to Copy: Controlling Speculative Decoding via Intrinsic Model Signals
SwitchSD uses internal LLM signals to adaptively switch between neural drafting and context-based copying in speculative decoding, boosting throughput up to 15%.
Researchers introduce SwitchSD, an adaptive speculative decoding framework that treats copying as a latent control signal of the target LLM. Lightweight probes trained on internal representations identify genuine copy-intent with AUC above 0.99, allowing dynamic switching between neural drafting (EAGLE/EAGLE3) and context-based copying. Experiments across the Llama and Qwen families show throughput gains up to 15% over state-of-the-art baselines like EAGLE3 by eliminating false-positive copy triggers from accidental repetitions.
- Treats copying as a latent control signal detected from model internals
- Probes identify true copy-intent with AUC greater than 0.99
- Fixes false-positive triggers caused by surface n-gram repetition
- Up to 15% throughput gain over EAGLE3 on Llama and Qwen models
Full article177 words · extracted from arxiv.org · click to collapse
Speculative Decoding (SD) has significantly accelerated Large Language Model (LLM) inference, yet existing approaches face a fundamental tradeoff between two drafting strategies: neural drafting and context-based copying. Neural drafts (e.g., EAGLE3) provide robust performance across diverse text settings, while copy-based methods achieve higher speedups in copy-intensive regimes by generating candidates faster and exploiting long repetition spans for near-perfect speculation. We analyze existing copy-based methods and find that they are prone to accidental repetitions where surface-level n-gram overlap does not reflect a structural intent to copy, leading to false-positive triggers that ultimately degrade throughput. We introduce SwitchSD, an adaptive framework that treats copying as a latent control signal of the LLM. By training lightweight probes on the target model's internal representations, SwitchSD identifies genuine copy-intent with high precision (AUC > 0.99). This allows the system to dynamically switch between neural drafting (e.g., EAGLE) and context-based copying. Our results across Llama and Qwen families demonstrate throughput gains of up to 15% over state-of-the-art baselines like EAGLE3, effectively turning copying from a noisy heuristic into a principled, model-aware decoding regime.
Text extracted automatically; images, tables and formatting may be missing. Original: https://arxiv.org/abs/2609.20186