AFFECT trains world models on action-induced visual change
Original title: Learning What Actions Change: Action Effect Matching for Dynamics-Faithful World Models
On DROID, policy-success correlation with the real robot rises from 0.47 to 0.79, with up to 23% lower FVD.
The authors propose AFFECT, an auxiliary objective that trains world models on the distribution of visual changes caused by action perturbations. Temporally correlated noise is injected into actions from a policy trained on task-agnostic human teleoperation, and the model is trained so generated videos respond as recorded videos do, without repeated executions from the same start state. Experiments cover OGBench simulation and real DROID and ALOHA-2 setups. The paper reports more faithful rollouts and up to 8.3% higher planning task success.
Unconfirmed: Correlation, FVD, and planning gains are author-reported and not independently verified.
- The method does not require repeated executions from the same initial state.
- Pearson correlation on DROID rises from 0.47 to 0.79.
- FVD falls by up to 23% on DROID.
- Planning with AFFECT-trained models improves task success by up to 8.3%.
From the original article
1 min read at huggingface.coWorld models aim to predict environment dynamics under actions, enabling the prediction of action consequences without executing them in the real world. Yet despite producing increasingly realistic videos, current world models often fail to capture how small action changes affect future observations.
By John Won, Yonghoon Dong, Jinwoo Shin · huggingface.co
A short excerpt; the article and its images belong to the publisher. Summary, key facts and assessment above are AI-written from the article and can be wrong.