HRoPE encodes patch surfaces as slanted planes for multi-view attention
Original title: HRoPE: Homography Relative Positional Encoding
Plane parameters need no ground-truth geometry and can be learned from view synthesis alone or taken from foundation models.
HRoPE represents each image patch as a slanted 3D plane and injects the resulting homographies into transformer attention as relative transformations. The prior is meant to correct perspective distortion and align features across viewpoints, unlike encodings based only on camera frustums or ray-marched depth. On RealEstate10K, CO3Dv2, and DL3DV, the best variant outperforms the geometric positional encodings that were evaluated for novel view synthesis.
- The method follows the slanted support-window assumption from classical multi-view stereo.
- Plane parameters can be predicted end-to-end under a view-synthesis objective.
- They can also be derived from foundation models instead of ground-truth geometry.
- Ablations are reported alongside results on three multi-view datasets.
From the original article
1 min read at huggingface.co
Geometric Relative Positional Encoding has become the primary mechanism for injecting spatial inductive biases into multi-view transformers. Existing designs condition attention on camera-level frustums or ray-marched depth, associating cross-view tokens through the sensor geometry.
By Huachen Gao, Yang Yang, Xiaoqian Liang · huggingface.co
A short excerpt; the article and its images belong to the publisher. Summary, key facts and assessment above are AI-written from the article and can be wrong.