LGWM verifies GUI agent transitions by predicting next-screen representations
Original title: Right Screen, Wrong Transition: World Models as Verifiers for GUI Agents
On RSWT-BENCH, where screen-only detectors are at chance, the training-free score hits 0.987 AUC in 17 ms.
Authors argue GUI agent safety depends on whether a screen was the expected result of an action, not on the screen alone. They present LGWM, a decoder-free world model trained on 1.85M real GUI transitions that predicts the next screen's representation so verification is a vector comparison. On their RSWT-BENCH they report 0.987 AUC at 17 ms, matching strong closed-source VLMs, and 0.953 AUC separating harmful from benign mismatches.
Unconfirmed: Accuracy and latency figures are the authors' reported results, not independently confirmed.
- Trained without semantic annotation on 1.85 million real GUI transitions.
- Authors report 0.987 AUC at 17 milliseconds per decision.
- About ten AUC points above generative GUI world models, with far lower latency.
- Residual direction reaches 0.953 AUC separating harmful from benign violations.
- Prompted VLMs were near chance on that harmful-versus-benign split.
From the original article
1 min read at huggingface.co
A login screen that appears after a tap on Sign in is expected; the same screen after a tap on View order is an attack. For GUI agents, safety is therefore a property of the transition rather than of the screen, and a monitor that inspects only screens can be defeated by reusing a legitimate one. Judging a transition requires an expectation of what should have followed the action.
By Jiaming Zhang, Xuan Wang, Fuyao Zhang · huggingface.co
A short excerpt; the article and its images belong to the publisher. Summary, key facts and assessment above are AI-written from the article and can be wrong.