Interactive Visual Gym evaluates visual assistants in closed loop
Original title: Interactive Visual Gym: Evaluating Visual Interaction Models with Simulated Users in Interactive Worlds
Gemini 4 Argon finishes 53.4% of 315 scenarios in time; a human study finds 78.5% metric agreement.
The paper presents Interactive Visual Gym, a closed-loop framework that jointly simulates executable task environments and human users for visual interaction models. It includes simulators for cooking, household manipulation, and mobile interfaces, plus a persona-conditioned user harness that can make controlled errors. IVG-Bench has 315 scenarios. Gemini 4 Argon, the best model tested, completes 53.4% of tasks within the time limit and is often too slow to prevent errors. A proof-of-concept GRPO run improves Qwen3.5-4B success and interaction quality in the cooking environment.
Unconfirmed: Benchmark scores and the human-study agreement are reported by the authors and not independently verified here.
- The best model is weak on factual grounding and actionable guidance.
- A human study finds 78.5% agreement with the interaction-quality metric.
- Closed-loop GRPO training improves Qwen3.5-4B in the cooking environment.
- Offline replay benchmarks cannot capture how assistant replies change later user actions.
From the original article
1 min read at huggingface.coVisual interaction models that support users during ongoing physical tasks require evaluation in closed loop, where assistant responses can change subsequent user behavior and task outcomes. Existing benchmarks largely rely on offline replay and therefore cannot capture these effects.
By Chenhao Zheng, Zixian Ma, Kai Li · huggingface.co
A short excerpt; the article and its images belong to the publisher. Summary, key facts and assessment above are AI-written from the article and can be wrong.