VoT: Vision-of-Thought for Unified Multimodal Representation Alignment
Researchers propose Vision-of-Thought (VoT), a discrete visual-planning token layer between VLMs and diffusion transformers improving text-to-image semantic alignment.
VoT introduces a discrete visual-thinking layer between vision-language models and diffusion transformers, letting the VLM act as a multimodal planner that emits tokens describing objects and layouts before pixel generation. A specialized VoT tokenizer is trained with VLM alignment, feature reconstruction, and vector-quantization losses. Experiments show improved semantic alignment and a structured, interpretable interface for controllable generation.
- VoT tokens represent high-level visual plans like objects and layouts
- Tokenizer trained with closed-loop VLM alignment and vector-quantization objectives
- Improves semantic alignment over text-encoder-plus-diffusion baselines
- Offers interpretable intermediate representation for controllable generation
- Products
- Vision-of-Thought (VoT)
From the original article
1 min read at arxiv.orgCurrent text-to-image systems typically employ a "text encoder plus diffusion decoder" paradigm, in which text semantics directly modulate continuous latent noise. Despite their success, these methods lack an explicit, interpretable intermediate representation that effectively bridges high-level linguistic semantics and low-level visual signals.
By Jingxiang Sun · arxiv.org
A short excerpt; the article and its images belong to the publisher. Summary, key facts and assessment above are AI-written from the article and can be wrong.