SpecCompact accelerates LLM agent context compaction with slot-parallel speculation
Original title: SpecCompact: Accelerating LLM Agent Context Compaction with Slot-Parallel Semantic Speculative Verification
SpecCompact delivers 2.04-5.63x faster context compaction on AppWorld and τ³ benchmarks by parallel slot generation and target-model verification.
LLM agents require context compaction to handle growing histories but summarization introduces latency. SpecCompact observes that independent structured slots retain information and proposes slot-parallel semantic speculation. A small drafter generates slots in parallel while the target model verifies and repairs them; subsequent drafting uses updated prefixes. System optimizations further cut repeated processing and calls.
- Fine-tunes small drafter for parallel slot generation
- Target model verifies proposed lines and repairs rejected content
- 2.04-5.63x speedups over Hermes-style summarization across two target models
- 8.68x median speedup in Hermes user study
From the original article
1 min read at arxiv.orgLLM agents use context compaction to manage growing interaction histories, but summarizing long histories with large models can introduce substantial delays. Compaction therefore requires balancing information retention against latency. We observe that structured summary slots can retain information when generated independently.
By Yulu An · arxiv.org
A short excerpt; the article and its images belong to the publisher. Summary, key facts and assessment above are AI-written from the article and can be wrong.