Dual Exam scores multimodal agents and world models separately
Original title: Dual Exam: Controlled Evaluation of Multimodal Agents and World Models
GPT 6 Astra leads the Agent Index at 31.3, while GPT-Image-2 leads full-coverage world-model overall at 21.7.
Dual Exam evaluates multimodal agents under fixed environment responses and world models under fixed action interventions across physical, digital, social, and scientific domains. The agent track compares 10 configurations on 1,088 items in 15 reporting categories. The world-model track has 400 cases in eight scenarios, with 15 configurations fully covered and one covering six. A 150-case pilot of one coupled pairing recorded zero task successes. The paper says the two score sets are not directly comparable and that predictive value for coupled success is unestablished.
Unconfirmed: Whether separate scores predict coupled success or reciprocal learning is not established.
- Agent evaluation uses 1,088 items across 15 reporting categories.
- Fifteen world-model configurations have full coverage of eight scenarios.
- The coupled 150-case pilot recorded zero task successes.
- Authors report uneven performance and failures to preserve task-relevant state.
- Products
- Dual Exam
- AI models
- GPT 6 AstraGPT-Image-2
From the original article
1 min read at huggingface.co
World Models (WMs) can support Multimodal Agents (MAs) through simulated experience, while Agent trajectories can inform World Model refinement. Evaluating this interaction requires separate evidence about action selection and state evolution.
By Zhefan Rao, Meng Chu, Xuan Zhang · huggingface.co
A short excerpt; the article and its images belong to the publisher. Summary, key facts and assessment above are AI-written from the article and can be wrong.