GeoUG uses generated depth to improve spatial understanding
Original title: Learning and Reasoning with Geometry: How Generation Helps Spatial Understanding
Reported circular accuracies are 97.9% on CV-Bench-3D, 90.8% on 3DCueBench, and 94.2% on DA-2K.
The paper introduces GeoUG, a model that combines metric-depth and multi-view generation with spatial understanding through a multi-scale depth codec on a pretrained visual autoencoder. Joint training produced task-dependent gains, and generated depth helped most on fine-grained geometric questions in 3DCueBench. Adaptive interleaved reasoning drafts an answer, then uses reinforcement learning to decide whether generating geometric evidence is worth the cost. The authors say GeoUG reaches the reported circular accuracies with fewer generation calls than always generating.
Unconfirmed: Accuracies and generation-call savings are author-reported and not independently verified.
- Circular accuracy is 97.9% on CV-Bench-3D and 94.2% on DA-2K.
- 3DCueBench circular accuracy is reported at 90.8%.
- The decision to generate evidence is trained with reinforcement learning.
- A multi-scale depth codec enables metric-depth generation.
From the original article
1 min read at huggingface.co
Image and video generators can produce spatially consistent visual content, but how to use this capability to improve spatial understanding remains underexplored. We study this problem from two perspectives: geometric supervision during training and explicit geometric evidence during inference.
By Rui Yang, Yanwei Li, Sili Chen · huggingface.co
A short excerpt; the article and its images belong to the publisher. Summary, key facts and assessment above are AI-written from the article and can be wrong.