Paper generates 3D scenes in implicit latents from 2D models
Original title: Any 3D Scene is Worth 1K Tokens: 3D-Grounded Representation for Scene Generation at Scale
Diffusion transformers model scenes in a 3D latent from DINOv2, SigLIP2, and DA3 instead of redundant 2D views.
The paper argues that 2D multi-view and video diffusion waste capacity and hurt spatial consistency when used for scene-level 3D generation. It builds a high-dimensional implicit 3D latent from 2D foundation models and runs diffusion transformers directly in that space. Authors report gains in efficiency and consistency and analyze how the 3D representation and model capacity scale.
Unconfirmed: Efficiency and consistency gains are reported by the authors, not independently verified here.
- 2D view latents are described as redundant and spatially inconsistent.
- The 3D latent is derived from DINOv2, SigLIP2, and DA3.
- Diffusion transformers operate directly in the 3D latent space.
- Authors also test scaling with representation and model capacity.
From the original article
1 min read at huggingface.co
Scene-level 3D generation has long been dominated by 2D multi-view or video diffusion models. Typically, these 2D-based approaches perform scene generation in a 2D latent space, which introduces two fundamental issues: (i) representing 3D scenes via 2D views leads to significant representation redundancy, and (ii) latent space rooted in 2D inherently limits the spatial consistency of the generated scenes.
By Dongxu Wei, Qi Xu, Zhiqi Li · huggingface.co
A short excerpt; the article and its images belong to the publisher. Summary, key facts and assessment above are AI-written from the article and can be wrong.