Unlocking Structured Pruning for Reasoning Language Models via Attention-Preserving MLP Trimming
LayerTrim prunes reasoning LLMs without training, cutting cost while preserving long-form accuracy.
LayerTrim is a training-free structured pruning method that removes redundant decoder-layer MLP features while preserving attention and depth for long-form reasoning. Across three reasoning models and five benchmarks, it outperforms existing structured pruning methods in most settings, improving average accuracy by up to 21.4 percentage points over RESP. On Qwen3-14B it delivers 1.24x faster prefill and 19.7% lower peak GPU memory than the dense model while retaining stronger reasoning accuracy and shorter generations.
- LayerTrim is training-free structured pruning that keeps attention and decoder depth.
- It beats prior pruning methods by up to 21.4 points over RESP.
- On Qwen3-14B, prefill is 1.24x faster and peak GPU memory falls 19.7%.
- Pruned models keep highest average accuracy with the shortest generation length.
From the original article
1 min read at huggingface.co
Structured pruning reduces LLM inference costs by removing model components to produce smaller dense models. However, existing methods can cause reasoning performance to collapse on long-form reasoning tasks such as AIME. We propose LayerTrim, a training-free structured pruning method that identifies redundant decoder layers and selectively trims their MLP features while preserving attention and decoder depth.
By Woosang Lim, Haneul Yoo, Vincent-Daniel Yun · huggingface.co
A short excerpt; the article and its images belong to the publisher. Summary, key facts and assessment above are AI-written from the article and can be wrong.