iPrefBench tests when mobile agents should infer or ask
Original title: iPrefBench: A User-Centric Benchmark for Personalized Mobile Use Agents
Seven frontier agents hit up to 91.6% action completion but only 47.7% full task success.
Researchers introduced iPrefBench, a benchmark for whether personal mobile-use agents infer missing details from history or ask the user. It runs in an executable iOS environment with an interactive simulated user, a persona of 3,051 past activity events, and 130 underspecified tasks across 26 apps. Each omitted requirement is either history-inferable or ask-only. Evaluated agents completed actions well but lagged on personalized task success, especially when they needed to request ask-only information.
- The benchmark scores both personalized task completion and user-agent interaction.
- Agents failed most on asking for information only the user can supply.
- Authors position iPrefBench as a testbed for agents that work with a specific user.
From the original article
1 min read at huggingface.co
An agent that truly knows and acts on behalf of its user has been a long-standing promise of AI assistants. Computer-use agents bring this promise within reach by capably executing fully specified tasks across apps. However, real-world user requests are often underspecified.
By Yuting Ning, Ruijia Cheng, Sushobhan Ghosh · huggingface.co
A short excerpt; the article and its images belong to the publisher. Summary, key facts and assessment above are AI-written from the article and can be wrong.