AgSpec: Pushing the Limits of Retrieval-Based Speculative Decoding in Coding Agent Pipelines
AgSpec speeds coding-agent generation via retrieval and adaptive speculative drafts, up to 4.76× throughput.
AgSpec is a retrieval-based speculative decoding framework for coding-agent pipelines. It retrieves from session, workspace, and global corpora stored in the agent's emission format, and bounds draft length with an offline-profiled cap plus online adaptation from verification feedback. On two repository-level multi-agent coding benchmarks it outperforms five retrieval-based drafters and EAGLE-3 in most settings, raising throughput over autoregressive decoding up to 4.37× at batch size 1 and 4.76× at batch size 16. The gains also hold on benchmarks without a repository or multi-agent pipeline.
- Retrieves from session, workspace, and global corpora in the agent's emission format
- Sets an offline draft-length cap and adapts it from verification feedback
- Up to 4.37× throughput at batch size 1 and 4.76× at batch size 16
- Beats five retrieval drafters and EAGLE-3 in most evaluated settings
From the original article
1 min read at huggingface.co
Retrieval-based speculative decoding (SD) drafts tokens by copying continuations from existing text, which suits coding agents that repeatedly reproduce code, logs, and earlier attempts.
By Sumin Lee, Sukmin Cho, Suengjae Lim · huggingface.co
A short excerpt; the article and its images belong to the publisher. Summary, key facts and assessment above are AI-written from the article and can be wrong.