ACT3D uses reinforcement learning so VLMs explore 3D from one view
Original title: Learning to Think and Explore in 3D World
Average accuracy reached 53.2% on six 3D benchmarks, 22.0 points above base Qwen and above Gemini 3.6 Flash.
The model keeps a vision-language model's single-image encoding and treats 3D as an environment of actions and rendered observations. After trajectory-supervised finetuning, reinforcement learning rewards final-answer correctness without specifying which viewpoints to visit. ACT3D uses 50-70% fewer input frames than existing open-source 3D methods and gains 5.9% on average on unseen domains versus its supervised initialization. Gains over the base model rise from 0.6% when the first view suffices to 16.3% when extensive exploration is required.
Unconfirmed: Benchmark scores are reported by the paper's authors.
- Each rollout starts from a first-person view and a visual question.
- The model may navigate and reorient before committing to a final answer.
- Analysis says it searches for occluded targets and skips unnecessary exploration.
- Reinforcement learning gives no direct supervision of exploration actions.
- AI models
- ACT3DQwenGemini 3.6 Flash
From the original article
1 min read at huggingface.co
We study how vision-language models (VLMs) can learn to think and actively explore in 3D space. Instead of providing multi-view images, videos, or 3D representations to the model, we preserve the native visual encoding of VLMs, i. e.
By Yijiang Li, Bingyang Wang, Nuno Vasconcelos · huggingface.co
A short excerpt; the article and its images belong to the publisher. Summary, key facts and assessment above are AI-written from the article and can be wrong.