RenderFormer-V2: Neural Rendering with Heterogeneous Scene Primitives
RenderFormer-V2 is a transformer-based neural renderer handling caustics, volumetric scattering and out-of-distribution materials without per-scene training or specialized code.
RenderFormer-V2 is a learned transformer-based neural rendering model that models global light transport as a sequence-to-sequence transformation, handling caustics, volumetric scattering, environment lighting, textured and displaced surfaces and out-of-distribution materials. It uses a two-stage process: a view-independent stage resolving primitive-to-primitive transport, and a view-dependent stage converting the neural scene representation into pixels. Improvements include combined windowed-attention with a rendering-informed attention sink for scalability, support for heterogeneous primitives like environment maps and participating media, and a surface-reflectance-independent neural material encoding, validated across diverse scenes with extensive ablations.
- Models global light transport as sequence-to-sequence transformation
- Two-stage design separates view-independent transport from view-dependent pixelization
- Windowed attention plus rendering-informed attention sink improves scalability
- Supports environment maps, participating media and BRDF-agnostic material encoding
Full article168 words · extracted from huggingface.co · click to collapse
We present 'RenderFormer-V2', a unified learned transformer-based neural rendering model, complementary to modern physics-based rendering systems, that can handle diverse light-transport effects such as caustics, volumetric scattering, environment lighting, textured and displaced surfaces and out-of-distribution materials without per-scene training or specialized code. RenderFormer-V2 models global light transport as a sequence-to-sequence transformation. Following its predecessor, RenderFormer-V2 also employs a two stage process: a view-independent stage that resolves intra-scene primitive to primitive transport, and a view-dependent stage that transforms the internal neural scene representation into image pixels. Different from RenderFormer, our model employs a novel combined windowed-attention and rendering-informed attention sink in the view-independent stage to improve scalability while maintaining render accuracy. To further improve versatility, RenderFormerV2 supports heterogeneous scene primitives, including environment maps and participating media, and it employs a material encoding independent of the underlying surface reflectance model that encodes material appearance via a novel neural embedding. We demonstrate the versatility of RenderFormer-V2 on a variety of scenes and perform an extensive ablation of the improved attention mechanism.
Text extracted automatically; images, tables and formatting may be missing. Original: https://huggingface.co/papers/2609.05738