arXiv introduces Visual Token Routing for efficient computer-use agents
Original title: Visual Token Routing: Learning to keep tokens that matter
VTR matches full-resolution performance on OSWorld and CUA-World using 2.5-5.8x fewer visual tokens, reducing inference cost by up to 3.3x.
The paper addresses computational cost of visual tokens in computer-use agents for long-horizon tasks. Most visual tokens are semantically irrelevant or temporally redundant. Visual Token Routing jointly trains vision-language models with a lightweight spatio-temporal router that decides which tokens enter the language model. With 2,000 training steps, VTR matches full model on OSWorld, CUA-World, and video game action prediction while using 2.5-5.8x fewer tokens. It outperforms resolution reduction or other token-dropping baselines by up to 12.8 points on OSWorld.
- matches full-resolution performance on OSWorld, CUA-World, and video games
- uses 2.5-5.8x fewer visual tokens
- reduces inference cost by up to 3.3x
- speeds up prefill by up to 2.9x
- outperforms other baselines by up to 12.8 points on OSWorld
From the original article
1 min read at arxiv.orgComputer-use agents (CUAs) have recently achieved superhuman performance on several long-horizon benchmarks. However, their computational cost limits their use in long-horizon tasks and real-time domains such as video games. A key bottleneck is representing each screenshot as thousands of visual tokens, which accumulate into long contexts as agents reason over extended interaction histories.
By Pranjal Aggarwal · arxiv.org
A short excerpt; the article and its images belong to the publisher. Summary, key facts and assessment above are AI-written from the article and can be wrong.