DLite drops draft-model history for speculative decoding on agentic workloads
Original title: DLite: History-free Speculative Decoding for Agentic AI
Authors report 2-6 acceptance length and gains of up to 70% over DSpark on agentic and long-context tests.
DLite is a history-free speculative decoding framework aimed at agentic workloads with long, changing contexts. It reuses the target model's contextual hidden states for local prediction instead of keeping a draft model's historical KV cache. A lightweight sequential head restores intra-block causal dependencies. The authors report 2-6 acceptance length and gains of up to 70% over DSpark.
Unconfirmed: Speed and acceptance figures are the authors' reported measurements.
- Existing draft models lose up to 80% of acceptance length on agentic workloads.
- Training uses a tailored two-stage strategy for the heterogeneous architecture.
- Tests cover general-purpose, long-context and agentic workloads.
From the original article
1 min read at arxiv.orgSpeculative decoding has become a mainstream technique for accelerating large language model (LLM) inference by using a lightweight draft model to generate candidate tokens that are subsequently verified by the target model. However, agentic workloads introduce long and continuously evolving contexts and frequent multi-turn interactions.
By Jiaming Xu · arxiv.org
A short excerpt; the article and its images belong to the publisher. Summary, key facts and assessment above are AI-written from the article and can be wrong.