Less Decoder is More Encoder: Geometric Representation Learning from Novel View Synthesis
SNAP, a self-supervised pose-conditioned transformer for novel view synthesis, learns transferable geometric representations competitive with geometry-supervised methods across five tasks.
The paper argues that encoder-based novel view synthesis (NVS) methods yield poor representations because spatially expressive decoders dilute the scene encoder and low-level pixel-space targets hinder feature learning. SNAP, a self-supervised encoder-decoder transformer, addresses both with a pose-conditioned local decoder and a latent-space reconstruction objective. It is competitive with special-purpose geometry-supervised methods and with self-supervised representations across visual localization, pose estimation, point correspondence, depth estimation, and robot manipulation. Its patch features show emergent viewpoint invariance approaching heavily supervised models with lower compute and data budgets, degrading more gracefully under camera shifts where standard 2D representations collapse.
- Spatially expressive decoders and pixel-space targets weaken NVS encoder representations.
- Pose-conditioned local decoder plus latent-space reconstruction fixes both issues.
- Competitive across localization, pose estimation, correspondence, depth, and robot manipulation.
- Emergent viewpoint invariance achieved with lower compute and data budgets.
From the original article
1 min read at arxiv.orgThis paper examines the role of Novel View Synthesis (NVS) in geometric representation learning. In principle, NVS should reason about 3D scene structure, thereby enabling transferable multi-view geometric representations. Yet, existing encoder-based NVS methods yield poor representations.
By Keerthi Kaashyap · arxiv.org
A short excerpt; the article and its images belong to the publisher. Summary, key facts and assessment above are AI-written from the article and can be wrong.