Imagine3D-LLM: Teaching MLLMs to Imagine 3D Scenes Before Answering
Imagine3D-LLM adds learnable summary tokens decoded into compact 3D Gaussian Splatting representations, improving multimodal LLM spatial reasoning across benchmarks.
Imagine3D-LLM is a multimodal LLM that appends a small set of learnable summary tokens after image tokens, decoding them into a compact 3D Gaussian Splatting scene representation supervised by a photometric reconstruction loss and trained jointly with next-token prediction. Inspired by human spatial reasoning, the model assembles a coarse 3D layout rather than relying on fine-grained pixel-level geometry cues. It consistently outperforms prior approaches on multiple spatial reasoning and 3D understanding benchmarks, and reconstruction supervision also induces stronger cross-frame correspondence in the model's image features.
- Learnable summary tokens decode to compact 3D Gaussian Splatting representations
- Trained jointly with photometric reconstruction loss and next-token prediction
- Outperforms prior methods on spatial reasoning and 3D benchmarks
- Reconstruction supervision induces cross-frame correspondence in image features
From the original article
1 min read at arxiv.orgReasoning about the 3D world from multi-view images remains a fundamental challenge for Multimodal Large Language Models (MLLMs). While modern MLLMs handle single-image inputs effectively, they struggle to integrate evidence across viewpoints into a coherent 3D understanding.
By Jaewoo Jung · arxiv.org
A short excerpt; the article and its images belong to the publisher. Summary, key facts and assessment above are AI-written from the article and can be wrong.