AffordanceWAM uses affordance fields to transfer human video to robot actions
Original title: AffordanceWAM: Affordance-Aware Joint World-Action Modeling for Robot Manipulation
A video diffusion Transformer jointly predicts future RGB, scalar affordance, heatmaps, and continuous actions without human action labels.
AffordanceWAM is a generative world-action model that represents object-centric spatiotemporal affordance as a scalar field and a heatmap inside predicted future video. Separate World and Action experts, coupled by masked joint self-attention, predict future RGB, affordance, and continuous robot actions under flow matching. Human video supervises the world streams; robot trajectories add action labels, so transfer does not need human action labels or retargeting. The authors report gains on RoboCasa, CALVIN ABC to D, and real-world manipulation, with RoboCasa improving as annotated human video scales.
- Human videos supervise RGB, scalar affordance, and heatmap streams only.
- Robot trajectories supply action supervision under a unified flow-matching objective.
- Under fixed robot data, RoboCasa scores rose as annotated human video increased.
- Evaluations include RoboCasa, CALVIN ABC to D, and real-world manipulation.
- AI models
- AffordanceWAM
From the original article
1 min read at huggingface.co
Generalizable robot manipulation requires predicting how a scene will evolve, identifying where interactions are feasible, and determining how to act. Action-labeled robot videos directly supervise control but are costly and limited in diversity, whereas egocentric human videos capture diverse interactions but lack robot actions and differ in embodiment and appearance.
By Jiadi You, Qize Yu, Yue Chen · huggingface.co
A short excerpt; the article and its images belong to the publisher. Summary, key facts and assessment above are AI-written from the article and can be wrong.