Paper proposes cache engineering so agents edit KV cache in place
Original title: Cache Engineering for Agents
A rolling-drop policy raised benchmark scores 1.7 points and cut cost 27.7% versus summarization.
Researchers argue that trimming or summarizing an agent trajectory invalidates later KV cache and forces re-prefilling. They propose cache engineering: drop excludes selected cache from future attention and repos adjusts position encodings, without changing the trajectory. Their AKV engine keeps a stateless API while managing stateful reuse, eviction, and restoration. Code is at github.com/thu-nics/AKV.
- Identical token sequences can have different cache states after different operations.
- AKV reached up to 1.2x output throughput and 1.5x faster P95 latency.
- Recursive policy improvement cut cost another 7.1% versus expert design.
- History dependence makes prefix matching insufficient for correct cache reuse.
- Products
- AKV
From the original article
1 min read at arxiv.orgLarge Language Model (LLM) agents can handle increasingly complex tasks, navigating long trajectories of observations and actions. As these trajectories grow, key-value (KV) cache I/O and storage become the dominant cost. Trajectories can also exceed the LLM's maximum context length, requiring control over active cache growth.
By Tianyu Fu · arxiv.org
A short excerpt; the article and its images belong to the publisher. Summary, key facts and assessment above are AI-written from the article and can be wrong.