Image Classifiers are Efficient Self-Supervised Video Representation Learners
VideoMSN turns pretrained image ViTs into efficient video learners, cutting pretraining by up to 160x.
VideoMSN is a masked Siamese framework that learns video representations by treating frame grids as super images and aligning spatially and temporally masked views with a shared Vision Transformer. It avoids heavy 3D backbones and reconstruction decoders, starting from pretrained DINO-v3 and DeiT-v3 image encoders. The method reports state-of-the-art results on Kinetics-400, UCF101, and HMDB51 while using up to 32x and 160x fewer video pretraining epochs than prior self-supervised video methods. It also performs strongly in low-shot classification.
- Videos are encoded as super-image grids of sampled frames.
- Spatial and temporal masking feed a shared ViT with a masked Siamese loss.
- Starts from pretrained DINO-v3 and DeiT-v3 image encoders, without a decoder.
- Reports SOTA on Kinetics-400, UCF101, and HMDB51 with up to 160x fewer epochs.
Full article173 words · extracted from arxiv.org · click to collapse
We introduce VideoMSN, a Masked Siamese Network framework for efficient self-supervised spatio-temporal representation learning in videos. Instead of relying on heavy 3D architectures or reconstruction-based autoencoders for learning with unlabeled data, we repurpose standard image Vision Transformers by representing videos as super images which are grids composed of frames sampled from videos. From each super image, we construct two views: one with spatial patch masking and the other with temporal frame masking, ensuring no information leakage across frames. A shared Vision Transformer (ViT) encoder aligns their embeddings using a masked Siamese loss, capturing both motion and appearance cues without reconstruction. Our decoder-free formulation leverages an image foundation model towards efficient video representation learning. Starting from pretrained DINO-v3 and DeiT-v3 image encoders, VideoMSN achieves state-of-the-art performance on Kinetics-400, UCF101, and HMDB51 while requiring up to $32\times$ fewer and $160\times$ fewer video pretraining epochs compared to prior video self-supervised learning methods. Our proposed approach also shows strong performance in low-shot classification, confirming the transferability of the learned representations in a label-scarce scenario. Project Page: https://cvir.github.io/projects/videomsn.
Text extracted automatically; images, tables and formatting may be missing. Original: https://arxiv.org/abs/2609.40347