Paper turns the Platonic Representation Hypothesis into a JEPA theorem
Original title: Proving and Improving the Platonic Representation Hypothesis with JEPAs
A 109M-parameter text JEPA beat language models up to 72B on alignment with DINOv2 and CLIP image encoders.
The authors prove a Platonic Representation Theorem: separately trained JEPAs learn the same representation up to one rotation when positive pairs are good views of one scene. They treat next-token prediction as a JEPA with bad views, which they say explains weak language-vision alignment until scale. Validation spans 145k captions to texts of 940M web images. Good views needed 42 times fewer samples than next-token prediction for the same alignment, and one rotation turned separate encoders into a COCO captioner with CIDEr 0.838.
- A 109M text JEPA beat evaluated language models up to 72B parameters.
- On ImageNet-100, pair-free top-5 accuracy was 64% versus at most 56% for models up to 8B.
- The scaling law of alignment predicted unseen runs within seed noise.
From the original article
1 min read at huggingface.co
Whether models trained on different modalities converge to the same representation of the world is central to AI. The Platonic Representation Hypothesis (PRH) conjectures that they do as they scale, but without a proof it offers neither a guarantee nor a recipe. We turn the hypothesis into a theorem.
By Randall Balestriero, Yann LeCun · huggingface.co
A short excerpt; the article and its images belong to the publisher. Summary, key facts and assessment above are AI-written from the article and can be wrong.