GaussianRenderFormer renders 3D Gaussian splats without a rasterizer
Original title: GaussianRenderFormer: Transformer-based Rendering of 3D Gaussian Splats
On 300 unseen objects it scores 38.08 dB PSNR, versus 44.02 dB when rasterizing the same pruned splats.
GaussianRenderFormer is a transformer that takes a set of 3D Gaussians and a camera and outputs an image directly, with no rasterizer. Each Gaussian is a token in a view-independent encoder; a view-dependent decoder turns patches of camera rays into pixels by attending to those tokens. Trained on 26,820 Objaverse objects pruned to 20,000 Gaussians each, it renders 300 unseen objects at 38.08 dB PSNR against full unpruned splats, while rasterizing the same pruned input reaches 44.02 dB. Diagnostics point to a view-decoder bottleneck; telling the decoder each Gaussian’s projected footprint adds 1.4 dB.
Unconfirmed: The authors do not rule out under-training as a cause of the remaining quality gap.
- Training used Objaverse objects pruned to 20,000 Gaussians each.
- The authors say the quality gap is not a generalization failure.
- Supplying projected Gaussian footprints, which the decoder could infer, adds 1.4 dB.
- Code, trained models, and videos are released on the project page.
- AI models
- GaussianRenderFormer
From the original article
1 min read at huggingface.co
Can a neural network learn to render 3D Gaussian splats on its own, with no rasterizer in the renderer? We present GaussianRenderFormer, a transformer that takes a set of Gaussians and a camera and outputs the image directly. Each Gaussian becomes one token in a view-independent encoder, and a view-dependent decoder turns small patches of camera rays into pixels by attending to those tokens.
By Shahaf Valencia Levy, Sagie Benaim · huggingface.co
A short excerpt; the article and its images belong to the publisher. Summary, key facts and assessment above are AI-written from the article and can be wrong.