GTR: Gated Token Recurrence for Efficient Dense Prediction
GTR introduces a softmax-free recurrent vision backbone reaching 58.9 box AP on COCO with sub-2ms latency, targeting high-resolution dense prediction and edge deployment.
Gated Token Recurrence (GTR) replaces quadratic global softmax attention with gated linear attention, alternating spatial scan directions, and spatially enhanced SwiGLU blocks. Distilled from a detection-specialized DINOv3 teacher using only final-layer patch-token alignment with squared L2 loss, GTR-L achieves 58.9 box AP on COCO val2017 at 1.908 ms median batch-one latency on an RTX 4090 under compiled FP16 with Objects365 detector pre-training. The same backbone transfers to instance segmentation, pose estimation, oriented detection, semantic segmentation, and monocular depth estimation. A specialized chunkwise CUDA operator runs 4x faster than FLA v0.5.0 at 1.6K tokens, and TensorRT deployment on DRIVE AGX Thor achieves 2.282-8.769 ms median batch-one latency.
- GTR swaps global softmax attention for gated linear attention with alternating spatial scans.
- GTR-L reaches 58.9 box AP on COCO val2017 at 1.908 ms on RTX 4090.
- Distilled from DINOv3 using only final-layer patch-token alignment with squared L2 loss.
- Custom chunkwise CUDA kernel is 4x faster than FLA v0.5.0 at 1.6K tokens.
- TensorRT on DRIVE AGX Thor yields 2.282-8.769 ms latency for edge deployment.
- Tags
- vision-backbonedense-predictionlinear-attentiongated-recurrenceknowledge-distillationobject-detectionedge-deployment
From the original article
1 min read at huggingface.co
Self-attention-based vision backbones perform well on dense prediction, but the quadratic computational cost of global softmax attention limits their efficiency as image resolution increases. We introduce Gated Token Recurrence (GTR), a softmax-free recurrent vision backbone that combines gated linear attention, alternating spatial scan directions, and spatially enhanced SwiGLU blocks.
By Zhe Feng, Longfei Liu, Wei Liu · huggingface.co
A short excerpt; the article and its images belong to the publisher. Summary, key facts and assessment above are AI-written from the article and can be wrong.