FLAT: Resampling Image and Text into 1D Flexible-Length Aligned Transmodal Tokens for Retrieval and Generation
FLAT jointly trains a multimodal encoder with text-to-image and image-to-text decoders, producing flexible-length tokens that hit 83.1 GenEval on T2I after fine-tuning.
FLAT (Flexible-Length Aligned Transmodal representations) is a pre-training framework that jointly optimizes a shared multimodal encoder with T2I and I2T decoders, combining contrastive alignment with bidirectional cross-modal generative objectives. It maps visual and textual inputs into a unified continuous 1D sequence space and uses nested dropout over prefix-K tokens for dynamic output lengths. A single pre-training stage supports cross-modal retrieval and generation (71.1 GenEval), with task-specific fine-tuning reaching 83.1 GenEval on T2I, 40.5 BLEU-4 and 138.6 CIDEr on MS-COCO captioning, and strong Recall@5 on MS-COCO and Flickr30K.
- Single-stage pre-training jointly optimizes shared encoder and T2I/I2T decoders
- Nested dropout over prefix-K tokens enables variable-length outputs
- 83.1 GenEval T2I and 40.5 BLEU-4 captioning after fine-tuning
- Representations support interpolation, latent arithmetic, zero-shot composed retrieval
- Tags
- contrastive-learningcross-modal-retrievalimage-captioningmultimodalnested-dropoutrepresentation-learningtext-to-image
- Products
- FLAT
- Organizations
- Hugging Face
- AI models
- FLAT
From the original article
1 min read at huggingface.co
Traditional multimodal representation learning and generation are two stages: a contrastive or self-supervised visual encoder is trained first, followed by a separate downstream generative model. This setup bottlenecks generative performance behind frozen embeddings.
By Guangyu Sun, Shlok Kumar Mishra, Wentao Bao · huggingface.co
A short excerpt; the article and its images belong to the publisher. Summary, key facts and assessment above are AI-written from the article and can be wrong.