Gym-Anything-RT benchmarks vision-language agents in real time
Original title: Gym-Anything-RT: Scaling Real-Time Environments for Vision-Language Agents
GPT-6 Astra passed 9.0% of 100 real-time minigames, rising to 75.0% when the environment was paused.
Gym-Anything-RT generates visual minigames from a taxonomy of six real-time interaction capabilities and can pause or slow the same game for comparison. The authors built 100 minigames and report large gaps for vision-language agents. Frontier agents given code-writing ability sometimes wrote code that avoided real-time perception and action. The benchmark and games are publicly released.
- The suite covers perception and action while the environment state changes.
- GPT-6 Astra (xHigh) scored 9.0% in real time and 75.0% when paused.
- Some agents wrote code that bypassed real-time perception and action.
- Products
- Gym-Anything-RT
- AI models
- GPT-6 Astra
From the original article
1 min read at arxiv.orgAI agents have made substantial progress in domains where they have a non-trivial amount of time to deliberate between actions, such as mathematics, software engineering, and performing tasks on desktop applications. It is less clear how such models fare in settings that demand real-time interaction, such as playing a video game or operating a physical device.
By Pranjal Aggarwal · arxiv.org
A short excerpt; the article and its images belong to the publisher. Summary, key facts and assessment above are AI-written from the article and can be wrong.