CacheRepair: Learning to Repair Cross-Chunk Context in RAG for KV Cache Fusion
CacheRepair restores cross-chunk KV context in RAG, speeding first token by 1.69-4.61x.
CacheRepair is a lightweight network that predicts the difference between independently computed chunk KV caches and caches from joint processing, then adds that residual to document-token caches. Each network is trained for a frozen target LLM on a generic retrieval corpus and reused across downstream datasets. On three LLMs and four datasets, it sits on the answer-quality-latency Pareto frontier in 11 of 12 combinations. The largest repairers deliver 1.69-4.61x median time-to-first-token speedups over full prefill and raise mean F1 by 2.1-26.1 percentage points versus direct cache reuse.
- Predicts residuals that restore missing cross-chunk KV context.
- Each repair network is trained for one frozen target LLM.
- On the quality-latency frontier in 11 of 12 settings.
- Largest repairers speed median TTFT by 1.69-4.61x over full prefill.
- Mean F1 rises 2.1-26.1 points versus direct cache reuse.
From the original article
1 min read at arxiv.orgMulti-document retrieval-augmented generation (RAG) requires a language model to process multiple retrieved text chunks before answering a question. Precomputing each chunk's KV cache independently and concatenating the caches when the chunks are retrieved can accelerate this step. However, the assembled cache lacks cross-chunk attention information, reducing answer quality.
By Genglin Wang · arxiv.org
A short excerpt; the article and its images belong to the publisher. Summary, key facts and assessment above are AI-written from the article and can be wrong.