ScoreCLR estimates contrastive repulsion with denoising score matching
Original title: Contrastive latent representation learning with denoising score matching
A lightweight score network replaces quadratic negative sampling; linear-probe accuracy stays close to SimCLR on CIFAR-10 and ImageNet-100.
The paper introduces ScoreCLR, a contrastive representation method that estimates the repulsive term from the score of the latent representation rather than a large batch of explicit negatives. Denoising score matching trains a lightweight network jointly with the encoder, with noise-level regularization and a Gaussian prior on the latent space. The repulsion cost is linear in batch size instead of quadratic. Under a shared recipe, linear-probe accuracy on CIFAR-10 and ImageNet-100 is close to SimCLR, while projector outputs have higher effective rank and sit closer to the Gaussian prior.
- Repulsion is estimated from the score of the latent representation.
- Cost scales linearly with batch size rather than quadratically.
- Compared with SimCLR under a shared training recipe.
- Evaluated on CIFAR-10 and ImageNet-100 linear probes.
- Projector outputs show higher effective rank than SimCLR.
From the original article
1 min read at arxiv.orgContrastive representation learning methods operate by pulling different views of the same data sample together while pushing them away from views of different samples. A fundamental challenge is the estimation of the repulsive force, which typically requires large batch sizes, can exhibit high variance, and is susceptible to the curse of dimensionality.
By Christopher Marouani · arxiv.org
A short excerpt; the article and its images belong to the publisher. Summary, key facts and assessment above are AI-written from the article and can be wrong.