AKV lets agents drop and reposition KV cache without re-prefill
Original title: Cache Engineering for Agents
A rolling-drop policy raises benchmark scores 1.7 points and cuts average cost 27.7% versus summarization.
The paper proposes cache engineering so LLM agents can change the active key-value cache without editing the trajectory prefix or re-prefilling retained context. Agents use drop to exclude selected cache from later attention and repos to adjust position encodings of kept keys. Because identical tokens can produce different cache states, the authors co-design the AKV serving engine for stateful reuse behind a stateless API. A tool-response rolling-drop policy beats threshold summarization on score and cost, and AKV reports higher throughput and lower P95 latency.
Unconfirmed: Speed and cost gains are reported by the authors for their benchmarks.
- Rolling-drop improves average scores by 1.7 points and cuts cost 27.7%.
- AKV reaches up to 1.2x output throughput and 1.5x faster P95 latency.
- Recursive policy self-improvement cuts cost another 7.1% versus expert design.
- Code is at github.com/thu-nics/AKV.
From the original article
1 min read at huggingface.co
Large Language Model (LLM) agents can handle increasingly complex tasks, navigating long trajectories of observations and actions. As these trajectories grow, key-value (KV) cache I/O and storage become the dominant cost. Trajectories can also exceed the LLM's maximum context length, requiring control over active cache growth.
By Tianyu Fu, Ruoxi Wang, Zihan Min · huggingface.co
A short excerpt; the article and its images belong to the publisher. Summary, key facts and assessment above are AI-written from the article and can be wrong.