SpecSlot cuts global attention cost in pixel-space transformers
Original title: Read, Allocate, Route: Role-Coupled Tokenization for Pixel-Space Generative Transformers
Only the lowest-frequency slot per spatial anchor attends globally; higher-frequency detail stays local.
The paper presents SpecSlot, a pixel-space rectified-flow Transformer that separates output detail from the length of the global attention sequence. Each spatial anchor expands into slots assigned disjoint DCT coefficient bands; only the lowest-frequency slot joins global attention, while other slots interact locally, and a spatial head adds a local correction. On ImageNet-256 the authors report FID comparable to a FLOPs-matched JiT control with faster convergence. On ImageNet-512 they report a 2.84 FID improvement at 3.9% fewer FLOPs and within 0.69 FID of JiT-B/16 at 38.2% lower cost. On text-to-image, SpecSlot beats JiT-B/8 on DPG-Bench by 1.06 at 40.6% lower forward cost.
- ImageNet-512 FID improves by 2.84 versus the control at 3.9% fewer FLOPs.
- SpecSlot is within 0.69 FID of JiT-B/16 at 38.2% lower cost.
- Text-to-image DPG-Bench score exceeds JiT-B/8 by 1.06 at 40.6% lower forward cost.
- Generative state, target, and loss stay in pixel space.
From the original article
1 min read at huggingface.co
Pixel-space generative Transformers typically use one token per image patch, so capturing finer detail requires more tokens and makes global attention increasingly expensive. We observe that finer detail does not necessarily require more global communication: low-frequency structure benefits from global exchange, while high-frequency detail can be modeled locally.
By Jiongfan Yang, Weichen Fan, Haiwen Diao · huggingface.co
A short excerpt; the article and its images belong to the publisher. Summary, key facts and assessment above are AI-written from the article and can be wrong.