Factorized vector quantization matches continuous visual tokens on understanding
Original title: Can Discrete Tokens Serve as Semantically Lossless Representations of Visual Signals?
The discrete model beats UniAR and Emu3-Chat by 22.8% and 59.2% on a nine-metric average.
The paper challenges the claim that quantization error makes discrete visual tokens semantically lossy relative to continuous representations. Using multimodal understanding as a proxy, it finds effective bit budget matters more than quantization error, and existing quantizers saturate as extra codebooks add less semantic information. Factorized Vector Quantization quantizes complementary fixed-dimensional views with symmetric parallel codebooks and keeps improving at larger budgets, matching continuous representations. Trained on industrial-scale multimodal data, the model is reported state of the art among discrete-representation models and competitive with Qwen3-VL, including fine-grained OCR. Code and models are released.
Unconfirmed: State-of-the-art status and the Qwen3-VL comparison are the authors' reported results.
- Discrete tokens with larger quantization error can still score better on understanding.
- Additional codebooks in existing quantizers add progressively less semantics.
- Authors call it the first discrete model competitive with strong continuous models such as Qwen3-VL.
- They also describe a practical recipe for reaching that competitiveness.
- Products
- Factorized Vector Quantization
From the original article
1 min read at huggingface.co
Discrete visual tokens represent vision in a discrete form analogous to language, offering a native interface for multimodal modeling under a shared next-token prediction objective. However, quantization error is widely assumed to cause unavoidable semantic loss, fundamentally limiting the semantic fidelity of discrete visual tokens and positioning continuous visual representations as an intrinsic upper bound.
By Jialv Zou, Xiaotong Li, Zhengrong Yue · huggingface.co
A short excerpt; the article and its images belong to the publisher. Summary, key facts and assessment above are AI-written from the article and can be wrong.