Holo4D steers video diffusion with reconstructed 4D trajectories
Original title: Holo4D: Holistic 4D Reconstruction as Geometric Control for Video Diffusion
Projected source-view trajectories condition a pretrained video model so unseen regions are filled without retraining the generator.
Holo4D is a video reshooting framework that synthesizes a monocular video along a new camera path while keeping observed geometry and motion. Feed-forward 4D reconstruction supplies trajectories that are projected into the requested views and encoded with geometry and source appearance to guide a pretrained video diffusion model. The scaffold plus the source video and target cameras grounds generation; unsupported regions fall to the generative prior. The authors say different 4D reconstructors can be swapped without retraining the video generator.
- The method targets novel-view synthesis and camera-controlled video reshooting.
- Conditioning uses reconstructed 3D trajectories as a source-view 4D scaffold.
- The video generator stays frozen when the reconstructor changes.
- Further results are referred to the project page.
- Products
- Holo4D
From the original article
1 min read at huggingface.co
Synthesizing a monocular video from a new camera trajectory requires preserving the observed scene geometry and motion while completing regions that become newly visible. Feed-forward 4D reconstructors recover the structure of the observed scene, whereas video diffusion models provide a generative prior for completing unseen content.
By Yushi Lan, Zeren Jiang, Koichi Namekata · huggingface.co
A short excerpt; the article and its images belong to the publisher. Summary, key facts and assessment above are AI-written from the article and can be wrong.