Halcyon trains text-to-image models without language pretraining
Original title: Text-to-Image Generative Modeling Without Language Pretraining
Authors say Halcyon-1.6B beats a JiT baseline on gFD with 275 times less pretraining compute.
Researchers introduce Halcyon, early-fusion multimodal autoregressive models pretrained from scratch on GPIC, a set of 100 million image-text pairs. They report that Halcyon-0.52B and Halcyon-1.6B beat named baselines at much lower compute, and that text-only pretraining did not improve generation. Image-to-text supervision also teaches captioning, and code and models are released.
Unconfirmed: Generation gains are the authors' reported benchmark comparisons, not an independent evaluation.
- GPIC contains 100 million image-text pairs.
- Halcyon-0.52B reports 48.0 gFD versus 63.3 for LlamaGen-XL/8.
- Halcyon-1.6B reports 35.3 gFD versus 41.0 for JiT-T2I-XXL/8++.
- The 0.52B comparison cites 18.2 times less pretraining compute.
- AI models
- HalcyonHalcyon-0.52BHalcyon-1.6B
From the original article
1 min read at huggingface.co
Modern text-to-image (T2I) systems use internet-scale language pretraining for text-conditioned image generation. We ask whether language pretraining is necessary for strong T2I generation. We introduce Halcyon, a family of early-fusion multimodal autoregressive models pretrained entirely from scratch on GPIC, a dataset of 100M image-text pairs.
By Keshigeyan Chandrasegaran, Yue Zhao, Armin W. Thomas · huggingface.co
A short excerpt; the article and its images belong to the publisher. Summary, key facts and assessment above are AI-written from the article and can be wrong.