Halcyon trains text-to-image models without language pretraining
Original title: Text-to-Image Generative Modeling Without Language Pretraining
The authors report Halcyon-1.6B beating a larger JiT baseline on gFD with 275 times less pretraining compute.
Researchers introduce Halcyon, a family of early-fusion multimodal autoregressive models trained from scratch on GPIC, a set of 100 million image-text pairs. They report Halcyon-0.52B beats LlamaGen-XL/8 on gFD with 18.2 times less pretraining compute, and Halcyon-1.6B beats JiT-T2I-XXL/8++ with 275 times less. Controlled tests found text-only pretraining did not improve text-to-image generation. Code and models are released by the authors.
Unconfirmed: The quality and compute comparisons are the authors' reported results, not an independent evaluation.
- Reported gFD is 48.0 for Halcyon-0.52B versus 63.3 for LlamaGen-XL/8.
- Halcyon-1.6B is reported at 35.3 gFD versus 41.0 for JiT-T2I-XXL/8++.
- Image-to-text supervision instead of caption dropout also teaches captioning.
- The authors say generation quality stayed competitive under the same pretraining budget.
From the original article
1 min read at arxiv.orgModern text-to-image (T2I) systems use internet-scale language pretraining for text-conditioned image generation. We ask whether language pretraining is necessary for strong T2I generation. We introduce Halcyon, a family of early-fusion multimodal autoregressive models pretrained entirely from scratch on GPIC, a dataset of 100M image-text pairs.
By Keshigeyan Chandrasegaran · arxiv.org
A short excerpt; the article and its images belong to the publisher. Summary, key facts and assessment above are AI-written from the article and can be wrong.