KV-cache repair budget should focus on target chunk
Original title: Where Should a KV-Cache Repair Budget Go? Distractor Structure and Contextual Compatibility in Non-Prefix Reuse
Recomputing the target chunk against reused cache outperforms other repair strategies in most settings.
Non-prefix KV-cache reuse encodes retrieved chunks independently and concatenates their caches. We vary chunks around the answer-bearing target across three 7B-8B models and four datasets. Reuse loses 19-100% of context-dependent questions that full prefill answers. Recoverable damage is spread over other chunks so restoring a few recovers little. The budget is better spent on the target through contextual compatibility. Recomputation of the target beats inserting its full-prefill KV in all seven settings tested. Question attention over reused caches selects the target in at most 18.8% of development questions. A retriever finds it more often and recomputing its top chunk after moving it next to the question significantly beats InfoFlow-style reordering at the same 5% budget in six of nine open-domain QA and API-documentation settings.
- Reuse loses 19-100% of context-dependent questions answered by full prefill
- Budget better spent on target chunk via contextual compatibility
- Recomputed target KV beats full-prefill insertion in all seven settings
- Retriever finds target more often than question attention in most cases
From the original article
1 min read at arxiv.orgNon-prefix KV-cache reuse encodes retrieved chunks independently and concatenates their caches. To find where a small repair budget should be spent, we vary the chunks around the answer-bearing one (the target) across three 7B-8B models and four datasets of dialogue memory, open-domain QA and API documentation.
By Yuxing Peng · arxiv.org
A short excerpt; the article and its images belong to the publisher. Summary, key facts and assessment above are AI-written from the article and can be wrong.