RoboQuest: Generalist Physical Agents that Search, Inspect and Test
RoboQuest finds frontier physical agents succeed on only 23% of exploration tasks.
RoboQuest is a benchmark for goal-directed embodied exploration in which agents must search, inspect, or test to gather missing task information and decide when a task is complete. It includes ten mobile manipulation tasks and evaluates five frontier multimodal agents plus a pi0.5 policy fine-tuned on released full-episode demonstrations. The best agent succeeds in only 23% of episodes, while the fine-tuned policy almost never succeeds. Isolated skill tests show execution is usually possible; failures are mostly premature commitment and unrepaired exploration side effects.
- Ten mobile manipulation tasks cover search, inspection, and interactive testing.
- Best frontier agent succeeds on only 23% of episodes.
- A fine-tuned pi0.5 policy almost never succeeds.
- Most failures come from stopping exploration before required evidence, not motor execution.
- Agents rarely prevent or repair disturbances caused by exploration.
Full article253 words · extracted from huggingface.co · click to collapse
Recent advances in multimodal foundation models have made them capable generalist physical agents for a range of manipulation tasks. However, successful operation in an unfamiliar environment may require an agent to seek task-relevant information through interaction when it is absent from the observations: it may need to determine where a relevant object is, inspect an unobserved property, or discover the effect of an unfamiliar tool. We thus introduce RoboQuest, a benchmark for goal-directed embodied exploration, where agents must actively acquire task-relevant information through physical interaction, use the resulting evidence to adapt subsequent actions, and autonomously decide when to commit to task completion. RoboQuest comprises ten mobile manipulation tasks centered on three forms of uncertainty: search, manipulation-based inspection, and interactive testing. We evaluate five frontier multimodal agents through a common visuomotor interface, as well as a π_{0.5} policy fine-tuned on the full-episode demonstrations we release. The best agent succeeds in only 23\% of the episodes, and the fine-tuned policy almost never succeeds. Isolated tests of the execution skills the tasks are built from, with the hidden information supplied, show that the agents can carry out most of the required actions, and our failure analysis attributes only a minority of the failures to execution. Our failure analysis further finds that the agents often stop exploring too early as they make decisions before observing the required evidence for task completion. We also find that agents rarely prevent or repair the disturbances caused by their exploration. Moreover, learning by trial and error remains difficult for most models.
Text extracted automatically; images, tables and formatting may be missing. Original: https://huggingface.co/papers/2610.10388