dNaViT encodes images as discrete general-purpose visual tokens
Original title: dNaViT: Discrete Codes are General-Purpose Visual Representations
Authors say it matches or beats task-specific baselines on 12 understanding, 4 generation, and one editing benchmark.
dNaViT is a Discrete Native-resolution Vision Transformer that turns visual features into discrete primitives analogous to language tokens. It starts from a frozen pretrained encoder and uses lightweight residual modulation plus grouped product quantization so discrete capacity grows without longer spatial sequences. The authors present it as a tokenizer for unified encoding and high-fidelity image decoding across understanding, generation, and editing.
Unconfirmed: Benchmark comparisons are reported by the authors and are not independently confirmed in this item.
- Built on a frozen pretrained encoder with a dual-branch objective.
- Grouped product quantization avoids longer spatial sequences.
- Evaluated on 12 understanding benchmarks and 4 generation benchmarks.
- Also tested on one image editing benchmark.
From the original article
1 min read at huggingface.co
Discrete autoregressive modeling, notably through next-token prediction, has become a cornerstone of modern language models. Extending this paradigm to vision allows visual and language models to converge on a unified framework, thereby fostering shared methodology and infrastructure.
By Xiaotong Li, Zhengrong Yue, Xu Huang · huggingface.co
A short excerpt; the article and its images belong to the publisher. Summary, key facts and assessment above are AI-written from the article and can be wrong.