PointWAM: 3D World Action Modeling for Dexterous Robotic Manipulation
PointWAM forecasts 3D scene and hand trajectories, lifting dexterous robot success on DexJoCo tasks.
PointWAM is a 3D world action model that represents scenes and hands as point trajectories in a shared space-time frame, then retargets predicted hand motion to robot actions. It pretrains on large-scale human demonstration videos without task-specific object or keypoint selection. Human-video pretraining raises average DexJoCo success by 56.9 percentage points, and scene-trajectory supervision adds 10.9 points. With both, it surpasses prior state of the art on ten DexJoCo tasks by 11.7 points and outperforms strong vision-language-action models on a real robot.
- Forecasts scene and hand motion as 3D point trajectories
- Pretrains on human videos without object or keypoint labels
- Human-video pretraining adds 56.9 DexJoCo success points
- Beats prior state of the art on ten DexJoCo tasks by 11.7 points
From the original article
1 min read at huggingface.co
World action models jointly learn to forecast world dynamics and predict robot actions, such that the learned internal world dynamics guide accurate actions.
By Chunghyun Park, Beomjun Kim, Seungcheol Park · huggingface.co
A short excerpt; the article and its images belong to the publisher. Summary, key facts and assessment above are AI-written from the article and can be wrong.