LeRF: Learning Reference Coordinate Frames for Perspective Taking Reasoning
LeRF trains vision-language models to build explicit reference frames for perspective-taking spatial reasoning.
LeRF trains vision-language models to build explicit reference coordinate frames for perspective-taking, where relations must be judged from another entity's viewpoint rather than the camera. Given an image and query, the model decides whether a frame is needed, grounds the reference entity, and predicts the frame origin and orientation. A lightweight renderer overlays the frame so later reasoning uses those visual cues without external perception models or 3D reconstruction. Supervised fine-tuning teaches tool use and frame prediction, followed by reinforcement learning on spatial visual question answering, and the method beats its backbone on perspective-taking benchmarks.
- VLMs often answer perspective questions from the camera viewpoint.
- LeRF predicts an entity-centered frame and overlays it on the image.
- Training uses supervised fine-tuning, then reinforcement learning on spatial VQA.
- It improves perspective-taking, frame grounding, and orientation estimation.
From the original article
1 min read at huggingface.co
Perspective taking is a fundamental component of spatial intelligence, requiring models interpret spatial relations from a specified viewpoint, such as that of another entity or an imagined observer.
By Bang Xiao, Wenqi Jia, Ozgur Kara · huggingface.co
A short excerpt; the article and its images belong to the publisher. Summary, key facts and assessment above are AI-written from the article and can be wrong.