I-JEPA predictor width decides which shared image factors stay readable
Original title: Shared Is Not Enough: Predictor Width and Factor Readability Under Masked Latent Prediction
On 0.7M-parameter encoders, a half-width I-JEPA predictor scored about 40% of SimCLR, and matching encoder width reversed that at every seed.
Masked latent prediction methods such as I-JEPA are meant to learn predictable features, but which factors they keep is unclear. Researchers trained small encoders with I-JEPA, SimCLR and BYOL on identical view pairs inside a simulator where every image factor is known, then probed a factor shared by both views. The usual narrower predictor retained less of that factor than the contrastive methods; widening it changed the ranking. The simulator and trained models are released.
- Encoders had 0.7 million parameters.
- The default predictor used half the encoder width and depth, matching ViT-B proportions.
- Its median probe score was about 40 percent of SimCLR.
- Full-width predictors beat both SimCLR and BYOL at every training seed.
From the original article
1 min read at arxiv.orgMasked latent prediction methods such as I-JEPA aim to learn predictable features, but what determines which information they retain is less clear. We built a simulator in which every factor behind an image is known, so a probe can measure how readable each factor remains after training. We trained small encoders (0.
By Nilotpal Kakati · arxiv.org
A short excerpt; the article and its images belong to the publisher. Summary, key facts and assessment above are AI-written from the article and can be wrong.