Copy the Same, Distill the Difference: Initializing Linear Vision Transformers
Copying MLP weights and distilling attention lets linear Vision Transformers match or beat pretrained Softmax ViTs.
Linear Vision Transformers replace softmax attention with linear-complexity operators but usually need costly from-scratch pretraining and lag softmax models. The authors find attention weights are operator-specific, so copying them can be worse than random initialization, while MLP weights transfer by direct copy. Distilling token-routing behavior together with copied MLPs closes the gap and can surpass softmax ViTs across variants, sizes, and datasets.
- Copying softmax attention weights into linear ViTs barely helps and can hurt.
- MLP weights are operator-agnostic and carry most of the pretrained benefit.
- Distilled attention plus copied MLPs can match or exceed softmax ViTs.
- Findings hold across linear ViT variants, model sizes, and datasets.
From the original article
1 min read at arxiv.orgLinear Vision Transformers (ViTs) are designed to replace the attention in Softmax ViTs with the linear-complexity attention operator for more efficient token routing, but they require from-scratch pre-training and typically underperform the original Softmax version. How to initialize linear ViTs both efficiently and effectively still remains unclear.
By Huaiyuan Qin · arxiv.org
A short excerpt; the article and its images belong to the publisher. Summary, key facts and assessment above are AI-written from the article and can be wrong.