GUI-JEPA scores mobile GUI actions in latent space without decoding
Original title: GUI-JEPA: Candidate-Conditioned JEPA Latents as a Fast Action Policy for Mobile GUI Agents
On AndroidControl it is 2.8–4.0x faster than a fine-tuned Qwen2.5-VL-7B agent while using 6% of the parameters.
Researchers propose GUI-JEPA, a decoder-free mobile GUI agent policy built on a Joint-Embedding Predictive Architecture. A pretrained JEPA encoder turns a screen into one latent per candidate action, and lightweight heads score all candidates against the instruction in one batched pass. On AndroidControl, the authors report 86–92% of a fine-tuned Qwen2.5-VL-7B agent’s in-distribution accuracy at 2.8–4.0x the speed on the same GPU. They also report that a randomly initialised predictor matches one pretrained on screen dynamics.
Unconfirmed: Speed and accuracy figures are the authors’ reported results, not independently verified.
- The policy selects actions without decoding any token.
- It uses 6% of the parameters of the compared 7B VLM agent.
- A random predictor scored as well as one pretrained on screen dynamics.
- Code is at github.com/bcwang-sjtu/GUI-JEPA.
From the original article
1 min read at arxiv.orgMobile GUI agents are typically fine-tuned 7B-scale vision-language models (VLMs) that generate each action as text. However, these VLMs decode every action token by token, and even agents that score candidate actions keep a large language model or VLM in the loop, making them slow and memory-hungry.
By Bingchao Wang · arxiv.org
A short excerpt; the article and its images belong to the publisher. Summary, key facts and assessment above are AI-written from the article and can be wrong.