Periodic Weak Spots: Phase Sensitivity from Chunked KV-Cache Compression
Chunked KV-cache compression creates phase-dependent retrieval gaps of up to 40 points in long-context models.
The paper finds that chunked KV-cache compression introduces a token phase relative to compression-window boundaries, and retrieval accuracy varies systematically with that phase. In large open-weight models, long-context retrieval accuracy differed by up to 40 percentage points across phases. Pretraining transformers from scratch across multiple compression designs reproduced the effect, and causal interventions showed attention components specialize by source phase. The authors say evaluations must measure every compression phase because high average accuracy can conceal periodic weak spots.
- Phase is a token's offset inside the compression window.
- Retrieval accuracy varied by up to 40 percentage points.
- The effect reproduced across compression designs trained from scratch.
- Average scores can hide systematic positional failures.
From the original article
1 min read at huggingface.co
Chunked KV-cache compression reduces the memory and attention costs of long-context inference by compressing windows of consecutive tokens into fewer cache entries at a fixed stride. Such compression also introduces a new positional coordinate: a token's phase, or its position relative to compression-window boundaries.
By Xingyu Zhu, Pu, Yi · huggingface.co
A short excerpt; the article and its images belong to the publisher. Summary, key facts and assessment above are AI-written from the article and can be wrong.