IGG benchmark asks GUI agents to reveal hidden targets before clicking
Original title: IGG: A Benchmark for Interactive GUI Grounding under Visibility Constraints
Humans score 94.6%, GPT-5.5 79.4%, and the best open-source VLM only 31.6% on 601 tasks.
Standard GUI grounding assumes the target is already visible. Interactive GUI Grounding has 601 tasks across 16 controllable environments and seven visibility constraints; the agent must make the target visible, then click it. Hinting Qwen3-VL which interaction to use lifts accuracy only from 30.8% to 33.9%, while performing the interaction for the model raises it to 81.9%. The authors say the gap is executing the reveal step and that the skill is learnable.
- Allowed actions are scrolling, moving, dragging, waiting, and clicking.
- Doing the reveal interaction for Qwen3-VL reaches 81.9% accuracy.
- The study covers seven visibility-constraint sub-types.
- Visibility recovery improved consistently across model families in their tests.
From the original article
1 min read at arxiv.orgGUI grounding benchmarks ask whether an agent can localize a target that is already visible in a static screenshot. Real interfaces often violate this assumption: a target may be off-screen, appear only after a state change or hover, remain occluded, or become actionable after a delay. This leaves the ability to recover target visibility unevaluated.
By Kyeongseon Kim · arxiv.org
A short excerpt; the article and its images belong to the publisher. Summary, key facts and assessment above are AI-written from the article and can be wrong.