DepthWorld: 3D World Model for Robot Manipulation
DepthWorld jointly predicts multi-view RGB and metric depth for robot manipulation from calibrated 3D data.
The paper presents a calibration pipeline that combines learned stereo depth with a joint factor graph, recovering a robot's shared kinematics and per-scene extrinsics from pooled episodes. Applied to DROID, it yields DROID-3D, with dense metric depth and recalibrated multi-view extrinsics below 0.7 pixel reprojection error on 90% of external-camera episodes. DepthWorld, built on Stable Video Diffusion, jointly predicts multi-view RGB and depth through spatial latent tiling while leaving the pretrained VAE unchanged. Depth supervision improves RGB prediction by 1.48 dB PSNR over an identical RGB-only baseline at the same training budget.
- A factor-graph pipeline calibrates shared robot kinematics and scene extrinsics.
- DROID-3D stays under 0.7 px error on 90% of external-camera episodes.
- DepthWorld predicts multi-view RGB and depth without changing the pretrained VAE.
- Depth supervision improves RGB PSNR by 1.48 dB versus an RGB-only baseline.
- AI models
- DepthWorldStable Video Diffusion
From the original article
1 min read at arxiv.orgWorld models offer a data-driven alternative to traditional simulators for robotics, with applications spanning policy evaluation, improvement, and planning. All of these uses depend on faithful 3D geometry, yet current video-based world models are trained on RGB alone and produce rollouts that look correct frame-by-frame but do not compose into a consistent 3D world.
By Jai Bardhan · arxiv.org
A short excerpt; the article and its images belong to the publisher. Summary, key facts and assessment above are AI-written from the article and can be wrong.