Your Transformer Is Secretly Elastic
A retraining-free method makes pretrained transformers elastic, pruning smoothly to 60% sparsity with small accuracy loss.
The paper introduces a retraining-free way to turn a pretrained transformer into one elastic model that adapts across computational budgets. It jointly ranks individual MLP neurons and attention heads by minimizing loss under soft pruning at many sparsity targets at once. On 27 encoders from 22M to 8B parameters, degradation stays smooth to 60% sparsity; pruning DINOv3 ViT-H+/16 to 50% removes 427M parameters with only a 0.8% ImageNet-1k linear-probe drop, and a 60% sparse AugReg ViT-B/16 reaches up to 1.80x throughput and 1.26x lower single-query latency on an A100.
- Retraining-free method ranks every MLP neuron and attention head jointly.
- Tested on 27 vision and text encoders from 22M to 8B parameters.
- DINOv3 ViT-H+/16 at 50% sparsity drops ImageNet accuracy only 0.8%.
- 60% sparse AugReg ViT-B/16 is up to 1.80x faster on an A100.
- Elastic models are built in minutes, with sub-linear construction time.
- Vendors
- NVIDIA
- Products
- NVIDIA A100
- AI models
- DINOv3 ViT-H+/16AugReg ViT-B/16
From the original article
1 min read at huggingface.co
Scaling up transformers has produced models with broad capabilities, deployed across heterogeneous hardware and fluctuating workloads. Yet, model families offer only a few fixed sizes (e. g.
By Walter Simoncini, Dheeraj Varghese, Gertjan J. Burghouts · huggingface.co
A short excerpt; the article and its images belong to the publisher. Summary, key facts and assessment above are AI-written from the article and can be wrong.