KV-streams for Efficient Compaction in Agentic Reinforcement Learning
KV-streams stream the KV cache through compaction, speeding agentic RL training 2.6 to 5 times without hurting performance.
KV-streams is a plug-and-play method for trainable context compaction in agentic LLM reinforcement learning. Instead of flushing the KV cache after each compaction, it streams the cache forward, cutting repeated prefills. Across three compaction strategies it reports a 2.6x to 5x wall-clock training speedup with no evidence of worse performance. The streamed cache can also act as a recurrent state, and a controlled experiment finds reinforcement learning alone is enough for that behavior to emerge.
- KV cache is streamed forward instead of flushed after compaction.
- Three compaction strategies train 2.6x to 5x faster.
- Streamed cache carries information long gone from the context.
- Controlled experiments show RL alone can induce that recurrent behavior.
From the original article
1 min read at arxiv.orgScaling the horizon of agentic LLMs is bottlenecked by the need to fit ever longer context traces in GPU memory. Context compaction has been the most popular mechanism to alleviate this issue, keeping GPU memory constant for a given trace. Unfortunately, most compaction strategies rely on prefilling the LLM context many times over, hindering training throughput.
By Emiliano Penaloza · arxiv.org
A short excerpt; the article and its images belong to the publisher. Summary, key facts and assessment above are AI-written from the article and can be wrong.