GeoPair: Geometry-Preserving Cross-Layer Factorization for Training-Free Transformer Compression
GeoPair compresses transformers training-free by pairing compatible layers and sharing factorized dictionaries.
GeoPair is a training-free transformer compression method that sequentially selects compatible cross-layer weight pairings and shared-dictionary factorizations while preserving each layer's activation geometry. Combined with structured sparsity, it produces efficient decompositions without retraining. The authors report state-of-the-art results across architectures, scales, and modalities, outperforming independent structured decompositions and heuristic pairwise factorizations.
- Training-free method pairs compatible cross-layer projections.
- Shared dictionaries preserve each layer's activation geometry.
- Structured sparsity yields efficient weight decompositions.
- Reports better results than heuristic pairwise factorizations.
Full article140 words · extracted from huggingface.co · click to collapse
Transformer architectures exhibit cross-layer redundancies, yet post-training compression pipelines typically optimize layers in isolation or rely on heuristic grouping strategies that disregard layer-specific activation geometries. We introduce a principled, training-free framework that sequentially optimizes cross-layer weight pairings and shared-dictionary factorizations. Rather than forcing weights of adjacent layers to share a basis or heuristically merging activation statistics, our approach identifies structurally compatible projections and learns a shared representation that better preserves each layer's distinct calibration geometry. Coupled with structured sparsity, this yields highly efficient weight decompositions without sacrificing functional fidelity. Across diverse architectures, scales, and modalities, our method achieves state-of-the-art results, consistently outperforming independent structured weight decompositions and alternative pairwise weight factorizations, which operate under heuristic grouping strategies. By replacing heuristic engineering strategies with a convergent, optimization-driven pipeline, we establish a theoretically grounded foundation for scalable, transformer compression across different modalities.
Text extracted automatically; images, tables and formatting may be missing. Original: https://huggingface.co/papers/2609.25963