Generative Representation Learning for Scalable Latent Tokenization
A generative tokenizer avoids representation collapse and reaches FID-50K 2.18 with JiT-L.
The paper introduces self-supervised representation learning intended as a tokenizer for generative models, using a loss based on factorized pointwise mutual information estimated from a learned generative model. It reports avoidance of dimensional collapse and feature suppression in settings where DINO and LeJEPA fail, with tokenization improving as compute increases. Combined tokenizer and generator efficiency is measured against direct pixel diffusion, reaching an FID-50K of 2.18 with JiT-L. On ImageNet and GPIC, extrapolated scaling is said to match target representations in about 7x fewer FLOPs than DINOv2 and 8x fewer than WebSSL.
- A generative PMI objective supervises self-supervised latent tokenization.
- The method avoids collapse cases where DINO and LeJEPA fail.
- JiT-L reaches FID-50K of 2.18 versus direct pixel-space diffusion.
- Claimed savings are about 7x FLOPs versus DINOv2 and 8x versus WebSSL.
From the original article
1 min read at arxiv.orgWe introduce an approach for self-supervised representation learning designed to be used as a tokenizer for generative modeling. Our objective supervises a representation loss by factorizing the pointwise mutual information of the data, which we estimate by taking information differences between conditional predictions from a learned generative model.
By Alan Baade · arxiv.org
A short excerpt; the article and its images belong to the publisher. Summary, key facts and assessment above are AI-written from the article and can be wrong.