LILIE language-informed loss improves pixel-space text-to-image training
Original title: A Language-Informed Loss for Visual Generation
JiT models trained with LILIE cut GPIC gFD from 66.84 to 53.01 versus a pixel-only baseline.
The paper tests whether language-informed supervision can train visual generators better than pixel-space loss alone. LILIE is an invertible normalizing-flow image encoder trained jointly with a vision-language model so captions can be recovered from noised features. The resulting loss is used to train JiT text-to-image models. On GPIC, gFD falls by 14 points, from 66.84 to 53.01, against a pixel-only baseline. Code, encoders, and generative models are released.
- LILIE is trained with next-token prediction alongside a vision-language model.
- The encoder is invertible and described as lossless.
- The reported gain is 14 gFD points on the GPIC dataset.
- Authors release code, encoders, and generative models.
From the original article
1 min read at huggingface.co
End-to-end visual generation with pixel-space loss has recently achieved competitive performance without the need for learned image tokenizers. In this work, we study whether language-informed supervision can provide a richer training signal than pixel loss for visual generation. We introduce LILIE (pronounced "lily"), a Language-Informed and Lossless Image Encoder.
By Suchir Agarwal, Keshigeyan Chandrasegaran, Ziyu Chen · huggingface.co
A short excerpt; the article and its images belong to the publisher. Summary, key facts and assessment above are AI-written from the article and can be wrong.