LEMS and KFAC-SVD improve SVD compression of large language models
Original title: Advancing SVD-based LLM Compression via Layer-Wise Error Model Search
Zero-shot gains reach 4.8 points through 70B models, and KFAC-SVD lowers perplexity 15% on average.
The paper addresses global rank allocation that misses error propagation and Fisher-based SVD estimators that suffer rank collapse. LEMS uses a layer-wise error surrogate and integer linear programming to allocate rank. KFAC-SVD uses token-wise statistics to improve decomposition quality. Across Mistral, Qwen3, and Llama 3 families, LEMS beats existing search strategies and KFAC-SVD reduces perplexity.
- LEMS zero-shot accuracy gains reach 4.8 percentage points and extend to 70B.
- KFAC-SVD reports an average perplexity improvement of 15%.
- A project page and code are published at lems-svd.github.io.
From the original article
1 min read at arxiv.orgLow-rank SVD-based compression offers a powerful strategy to reduce the computational costs of LLMs. However, existing methods face two key limitations: (i) global rank allocation, where uncalibrated error proxies fail to capture complex error propagation, and (ii) decomposition quality, where Fisher-based estimators suffer from severe rank collapse.
By Moritz Thoma · arxiv.org
A short excerpt; the article and its images belong to the publisher. Summary, key facts and assessment above are AI-written from the article and can be wrong.