Your Transformer Is Secretly Elastic
A retraining-free method turns transformers into elastic models that prune smoothly up to 60% sparsity.
Researchers present a retraining-free method that turns a pretrained transformer into one elastic model spanning many compute budgets. Individual MLP neurons and attention heads are ranked jointly by minimizing loss under soft pruning across many sparsity targets at once. Across 27 vision and text encoders from 22M to 8B parameters, degradation stays smooth up to 60% sparsity and beats one-shot baselines under deep pruning. Pruning DINOv3 ViT-H+/16 to 50% sparsity drops 427M parameters with only a 0.8% ImageNet-1k linear-probing accuracy loss; a 60%-sparse AugReg ViT-B/16 reaches up to 1.80x throughput and 1.26x lower single-query latency on an A100.
- Retraining-free method ranks MLP neurons and attention heads jointly.
- Tested on 27 vision and text encoders from 22M to 8B parameters.
- DINOv3 ViT-H+/16 at 50% sparsity loses 0.8% ImageNet linear-probing accuracy.
- AugReg ViT-B/16 at 60% sparsity reaches 1.80x A100 throughput.
- Vendors
- NVIDIA
- AI models
- DINOv3 ViT-H+/16AugReg ViT-B/16
From the original article
1 min read at arxiv.orgScaling up transformers has produced models with broad capabilities, deployed across heterogeneous hardware and fluctuating workloads. Yet, model families offer only a few fixed sizes (e. g.
By Walter Simoncini · arxiv.org
A short excerpt; the article and its images belong to the publisher. Summary, key facts and assessment above are AI-written from the article and can be wrong.