Ego2Act: Evaluating Goal-Directed Manipulation in Egocentric Video Generation
Ego2Act tests whether video models can simulate multi-step egocentric manipulation from a scene image and goal.
Ego2Act is a goal-directed benchmark of 2,640 videos spanning 110 everyday tasks with varying clutter and multi-step complexity. Given an initial scene and a high-level goal, it tests whether video models can generate realistic egocentric hand manipulations that complete the task. The authors also introduce Ego2ActJudge, a reference-free evaluator that better matches human judgments of task completion and physics plausibility. Generated simulations often skip or only partly execute steps and fail at fine-grained object dynamics and persistent world state.
- Ego2Act covers 2,640 videos from 110 real-world multi-step tasks.
- Models skip or partially execute steps, so later states and goals fail.
- Fine-grained physics and persistent world modeling remain weak.
- Reference-free Ego2ActJudge aligns better with human judgments than baselines.
From the original article
1 min read at huggingface.co
Video generation models are increasingly being explored as world simulators for embodied planning and learning. To do so effectively, these models must not only generate visually appealing frames, but also predict how environments dynamically evolve when executing goal-directed actions.
By Patrick Amadeus Irawan, Iskandar Muda Rizky Parlambang, Rava Maulana · huggingface.co
A short excerpt; the article and its images belong to the publisher. Summary, key facts and assessment above are AI-written from the article and can be wrong.