Studying Image Tokenizers as Visual Languages in Unified Multimodal Models
A controlled pure-autoregressive testbed shows task-specific validation losses rank image tokenizers differently, with I2T loss the most consistent signal.
Researchers built a controlled pure-autoregressive testbed and tracked task-specific validation losses during multimodal continual pretraining across text, image, text-to-image (T2I), and image-to-text (I2T) prediction. They find losses should be analyzed per task because they exhibit distinct scaling behavior and rank tokenizers differently, and that the loss-performance relationship depends on the predicted token space. I2T loss, computed over a shared text vocabulary, correlates consistently with both generation and visual understanding performance after supervised finetuning. Case studies revisit the discriminator, semantic supervision, and vocabulary size as tokenizer design axes.
- Task-specific validation losses show distinct scaling behavior and rank image tokenizers differently.
- I2T loss over a shared text vocabulary gives the most consistent cross-tokenizer signal.
- I2T loss correlates with both generation and visual understanding performance after supervised finetuning.
- Better image reconstruction does not necessarily yield lower task losses or stronger downstream performance.
- Image tokenizer choice affects text modeling under joint multimodal optimization.
From the original article
1 min read at arxiv.orgImage tokenizers define the ``visual language'' of unified multimodal models, yet are commonly studied through isolated metrics or generation-/understanding-only evaluations. These evaluations do not fully capture how visual tokens behave when modeled jointly with text.
By Siting Li · arxiv.org
A short excerpt; the article and its images belong to the publisher. Summary, key facts and assessment above are AI-written from the article and can be wrong.