279M Urdu UI ranker beats 2B and 7B VLMs on referential grounding
Original title: Rank, Don't Regress: A 279M Text Ranker Beats 2B and 7B VLMs on Referential Urdu GUI Grounding
On 304 held-out instructions from six Urdu news sites, the DOM ranker scores 73.4% point-in-box accuracy versus 29.6% and 23.7% zero-shot.
Instead of regressing click coordinates from screenshots, the authors rank candidate DOM elements with a 279M-parameter mDeBERTa-v3-base cross-encoder and click the centre of the chosen box. A frozen set from six Urdu news sites keeps test domains out of training; 149 of 304 instructions are referential. The ranker leads both zero-shot VLMs and a string-matching baseline, with errors concentrated on identical text and ordinal instructions.
- On the referential slice the ranker scores 65.8%, against 30.2% for untrained string matching.
- A similar-size coordinate regressor trained on the same pages scores 1.6%.
- Adding element geometry as input tokens lifts the best variant to 75.7%.
- Zero-shot comparators are UI-TARS-2B at 29.6% and Qwen2.5-VL-7B at 23.7%.
From the original article
1 min read at arxiv.orgUI grounding asks a model to find the screen element that a user instruction refers to. Most current systems use a vision-language model (VLM) that reads a screenshot and outputs pixel coordinates. We study this task on Urdu mobile web pages, a setting with no published UI grounding results.
By Hamza Raziq Khan · arxiv.org
A short excerpt; the article and its images belong to the publisher. Summary, key facts and assessment above are AI-written from the article and can be wrong.