FocusGUI adapter cuts sequential distraction in omni-modal GUI agents
Original title: Focused Omni-Modal GUI Agents
Across four frozen backbones, authors report Type Match up 33.2 points and Exact Match 3.0 times the frozen baseline.
FocusGUI is a lightweight adapter for GUI agents that observe screenshots, audio, video, interaction history, and device state. It compresses step-relevant evidence into a latent and injects it into a frozen decoder without backbone fine-tuning. Training uses action supervision only. Authors report gains over frozen backbones and the strongest existing enhancement method.
Unconfirmed: The reported match-rate gains come from the authors and are not independently verified here.
- Sequential Distraction is described as compounding action-feedback errors into trajectory drift.
- The adapter needs no modality-relevance labels or backbone fine-tuning.
- Average Type Match rose by 33.2 and Exact Match by 3.0 times, authors say.
- Type Match also rose by 30.1 over the strongest existing enhancement method.
From the original article
1 min read at huggingface.co
Graphical User Interface (GUI) agents increasingly act over streaming omni-modal observations spanning screenshots, audio, video, interaction history, and device states. However, exposing this complete context uniformly at every step ignores that relevance is transient and sparse: the signal determining one action can become clutter at the next.
By Fuyao Zhang, Wenwen He, Jiaming Zhang · huggingface.co
A short excerpt; the article and its images belong to the publisher. Summary, key facts and assessment above are AI-written from the article and can be wrong.