ReaDy: Recurrent States as Dynamic Context for Speculative Drafting in Hybrid Linear Attention Large Language Models
ReaDy uses recurrent states for speculative drafting, reaching 3.16× speedup on Qwen3.8-27B.
ReaDy is a speculative-decoding method that uses hybrid linear-attention recurrent states as extra context while drafting tokens. Each draft position learns a query over the target model’s committed recurrent states, and a lightweight state reader folds that context into draft representations with residual updates, low-rank reads, and fused operators. Built on DSpark’s parallel backbone and sequential head, it uses a two-stage adaptation strategy. Across seven math, code, and dialogue benchmarks, it delivered an average 3.16× speedup over autoregressive decoding with Qwen3.8-27B on one NVIDIA B200, beating DSpark on every benchmark by 7.97% throughput and 8.50% acceptance length.
- ReaDy retrieves position-specific draft context from HLA recurrent states.
- A lightweight state reader applies residual updates with low-rank reads.
- It reaches 3.16× speedup for Qwen3.8-27B on one NVIDIA B200.
- Throughput rises 7.97% and acceptance length 8.50% over DSpark.
Full article205 words · extracted from arxiv.org · click to collapse
Speculative decoding accelerates autoregressive generation by drafting multiple tokens per step, yet feature-based drafters rely on target hidden features whose contextual emphasis may not align with the needs of subsequent draft positions. Hybrid linear attention (HLA) models maintain compact recurrent states that accumulate historical associations, offering an efficient source of complementary context for speculative drafting. We introduce ReaDy, the first speculative decoding method, to the best of our knowledge, to leverage HLA recurrent states for position-specific context retrieval during drafting. ReaDy equips each draft position with a learned query to retrieve supplementary context directly from the target's committed recurrent states. A lightweight state reader integrates this context into draft representations through residual updates, while low-rank state reads and fused operators keep additional computation and I/O overhead low. Building on DSpark's parallel backbone and sequential prediction head, ReaDy employs a lightweight two-stage adaptation strategy, with reader capacity, state sources, and residual placement examined through ablation studies. Across seven benchmarks spanning mathematics, code generation, and dialogue, ReaDy achieves an average $3.16\times$ decoding speedup over autoregressive decoding with Qwen3.8-27B on a single NVIDIA B200 GPU. It outperforms DSpark on every benchmark, improving throughput by 7.97% and acceptance length by 8.50% on average. Our code is available at https://github.com/StartLuxLabs/ReaDy.
Text extracted automatically; images, tables and formatting may be missing. Original: https://arxiv.org/abs/2610.20503