One Block, Multiple Depths: Recurrent Vision Transformers with Depth-Programmed Experts
A recurrent vision Transformer matches full-depth encoder accuracy with about 70% fewer stored parameters.
reViT applies a single Transformer block recurrently and programs each depth's feed-forward network as a convex combination of a shared expert bank using a normalized-depth coordinate. Trained from scratch, reViT-B/16 matches DeiT III accuracy on ImageNet-1k with about 70% fewer stored parameters. An eight-expert model distilled only from DINOv2 output features retains nearly all of the teacher's linear-probe accuracy and transfers to segmentation and depth prediction. Elastic-depth training lets one checkpoint operate at multiple depths.
- One recurrent block matches full-depth encoder accuracy at similar FLOPs.
- Weight-space merging beat token-dispatch and output-mixture MoE variants.
- reViT-B/16 matches DeiT III with about 70% fewer stored parameters.
- Eight-expert distillation nearly preserves DINOv2 linear-probe accuracy.
- AI models
- reViT-B/16DeiT IIIDINOv2
From the original article
1 min read at huggingface.co
In this work, we show that a single Transformer block, applied recurrently, can match the accuracy of a full-depth vision encoder at comparable inference FLOPs without intermediate feature distillation. reViT restores depth-specific transformations by representing the FFN at each recurrent depth as a convex combination of a small shared expert bank.
By Adrian Bulat, Yassine Ouali, Georgios Tzimiropoulos · huggingface.co
A short excerpt; the article and its images belong to the publisher. Summary, key facts and assessment above are AI-written from the article and can be wrong.