DepthGraft shares KV hierarchically to regularize deep LLMs
Original title: DepthGraft: Structural Regularization through Hierarchical Cross-Layer KV Reconstruction
On a 16B MoE model the authors report +7.0 points on MMLU-Pro while cutting KV cache by about one third.
DepthGraft is a hierarchical cross-layer key-value sharing design meant to counter depth-wise information dilution in Pre-LN language models. Deep-layer KV is reconstructed from stride-aligned shallow-middle source pairs, creating non-local gradient paths that the authors treat as structural regularization. The design keeps early prefill completion and reduces KV cache by roughly one third. Experiments on dense and MoE models from 0.5B to 16B parameters under GQA and MLA report consistent downstream gains, including 7.0, 6.6, and 6.4 percentage points on MMLU-Pro, CMMLU, and C3 for the 16B MoE model.
Unconfirmed: Reported benchmark gains are the paper's own measurements.
- Models tested range from 0.5B to 16B parameters, dense and MoE.
- Results cover both GQA and MLA attention variants.
- 16B MoE gains cited: 7.0% MMLU-Pro, 6.6% CMMLU, 6.4% C3.
- Project page: github.com/jianke0604/DepthGraft.
From the original article
1 min read at huggingface.co
Scaling depth is a key route to stronger large language models, but Pre-LN architectures often suffer from depth-wise information dilution, limiting the effective use of deep layers. While cross-layer connectivity has proven effective at mitigating this problem, existing designs usually introduce additional computation or memory access. We revisit cross-layer KV sharing as an efficient form of such connectivity.
By Xiaohan Qin, Xiangdong Zhang, Yu Wang · huggingface.co
A short excerpt; the article and its images belong to the publisher. Summary, key facts and assessment above are AI-written from the article and can be wrong.