GeoPair: Geometry-Preserving Cross-Layer Factorization for Training-Free Transformer Compression
GeoPair compresses transformers training-free by pairing compatible layers and sharing factorized dictionaries.
GeoPair is a training-free transformer compression method that sequentially selects compatible cross-layer weight pairings and shared-dictionary factorizations while preserving each layer's activation geometry. Combined with structured sparsity, it produces efficient decompositions without retraining. The authors report state-of-the-art results across architectures, scales, and modalities, outperforming independent structured decompositions and heuristic pairwise factorizations.
- Training-free method pairs compatible cross-layer projections.
- Shared dictionaries preserve each layer's activation geometry.
- Structured sparsity yields efficient weight decompositions.
- Reports better results than heuristic pairwise factorizations.
From the original article
1 min read at huggingface.co
Transformer architectures exhibit cross-layer redundancies, yet post-training compression pipelines typically optimize layers in isolation or rely on heuristic grouping strategies that disregard layer-specific activation geometries. We introduce a principled, training-free framework that sequentially optimizes cross-layer weight pairings and shared-dictionary factorizations.
By Baher Mohammad, Ammar Ali, Stamatios Lefkimmiatis · huggingface.co
A short excerpt; the article and its images belong to the publisher. Summary, key facts and assessment above are AI-written from the article and can be wrong.