Video-Point Model forecasts multi-view video and 3D point trajectories
Original title: Learning the 4D World through Point Trajectory Forecasting
Point supervision on VPM-Data and VP-WorldBench improves video fidelity, 3D motion, and robot manipulation success.
Researchers introduce Video-Point Model (VPM), a world model that forecasts future videos from arbitrary viewpoints and 3D trajectories of persistent scene points from multi-view histories and a language instruction. Shared point trajectories anchor appearance changes across cameras. VPM-Data and VP-WorldBench support training and evaluation, and action tokens extend the model to robot control.
- Camera-conditioned processing integrates complementary observations across views.
- A trajectory head predicts how physical points move through time.
- Action tokens are denoised jointly with future videos for robot control.
- Point supervision beat a video-only counterpart on VP-WorldBench and 4D benchmarks.
From the original article
1 min read at huggingface.co
Reliable multi-view forecasting requires videos from different cameras to describe the same evolving 3D scene. To this end, we introduce Video-Point Model (VPM), a world model that conditions on multi-view video histories and a language instruction to jointly forecast future videos from arbitrary viewpoints and the 3D trajectories of persistent scene points.
By Byungwoo Jeon, Sunghwan Hong, Dohyeon Kim · huggingface.co
A short excerpt; the article and its images belong to the publisher. Summary, key facts and assessment above are AI-written from the article and can be wrong.