WorldFlow Jointly Predicts Video and 3D Motion
Original title: WorldFlow: Joint Video and 3D Motion Prediction via Large-Scale Pretraining
WorldFlow reduces ADE by 54.2% on SpatialCog set versus baselines using a Mixture-of-Transformers backbone.
WorldFlow is a generative world model that jointly denoises video and query-based 3D trajectories via a unified MoT backbone under rectified-flow objective. It parameterizes 3D motion as continuous displacement tokens aligned with video grid. Trained on 1.44 million heterogeneous samples, it achieves lowest ADE and FDE across three evaluation sets including WorldFlowBench with 1,678 samples.
- Trained on 1.44 million object-level samples from simulated and real domains
- WorldFlowBench includes 1,678 samples from 11 data sources
- 54.2% ADE reduction on SpatialCog versus strongest baseline
- 55.9% ADE and 68.5% FDE reduction versus flow-only variant
From the original article
1 min read at huggingface.co
Predicting future physical states stands as a central challenge in embodied intelligence. Existing video world models leave 3D dynamics implicit, action-based formulations remain tied to specific robot configurations, and 3D motion prediction models use flow-only supervision, providing a limited learning signal for physical dynamics.
By Yuhao Lu, Jiongxu Chen, Hexiang Wei · huggingface.co
A short excerpt; the article and its images belong to the publisher. Summary, key facts and assessment above are AI-written from the article and can be wrong.