Studying Image Tokenizers as Visual Languages in Unified Multimodal Models

A controlled pure-autoregressive testbed shows task-specific validation losses must be analyzed per task, with I2T loss over a shared text vocabulary the most consistent signal, and image tokenizer choice can affect text modeling under joint multimodal…

Researchers built a controlled pure-autoregressive testbed to study image tokenizers as the 'visual language' of unified multimodal models, tracking task-specific validation losses during multimodal continual pretraining across text, image, text-to-image (T2I), and image-to-text (I2T) prediction. They find that losses should be analyzed per task because they exhibit distinct scaling behavior and rank tokenizers differently, and that the loss-performance relationship depends on the predicted token space. I2T loss, computed over a shared text vocabulary, gives the most consistent cross-tokenizer signal and correlates consistently with both generation and visual understanding performance after supervised finetuning. The study also shows that better image reconstruction does not necessarily yield lower task-specific losses or stronger downstream performance, and that tokenizer choice can influence text modeling under joint optimization. Case studies revisit the discriminator, semantic supervision, and vocabulary size as tokenizer design axes.

• Researchers built a controlled pure-autoregressive testbed to study image tokenizers as the 'visual language' of unified multimodal models.
• The testbed tracks task-specific validation losses during multimodal continual pretraining across four prediction tasks: text, image, text-to-image (T2I), and image-to-text (I2T).
• Validation losses exhibit distinct scaling behavior per task and rank image tokenizers differently.
• The loss-performance relationship depends on the predicted token space.
• I2T loss computed over a shared text vocabulary gives the most consistent cross-tokenizer loss-performance signal.
• I2T loss correlates consistently with both generation and visual understanding performance after supervised finetuning.
• Better image reconstruction does not guarantee lower task-specific losses or stronger downstream performance.
• Image tokenizer choice can affect text modeling under joint multimodal optimization.