PersonaCUABench: computer-use agents lose 36% completion on persona requests
Original title: PersonaCUABench: Benchmarking Computer-Use Agents Serving Diverse Personas
Nine backbones ran 3,330 times; top satisfaction after full persona completion was 62.7% versus 84.1% on original requests.
PersonaCUABench keeps 74 tasks across 11 domains fixed and varies only who asks, using 44 personas in four groups. A persona agent rewrites each instruction and answers questions; the computer-use agent never sees the profile or original task. Across nine backbones and 3,330 runs, mean task completion falls 36% relative to original requests. The authors argue agents should be scored on expressed requirements and constraints, not completion alone.
- The benchmark uses 74 tasks, 11 domains, and 44 personas in four groups.
- In 59.9% of cells that lose points, the agent engages and still fails.
- User-requested stops are 11.2% of point losses the checker scores as failures.
- Full persona completions earn top satisfaction in 62.7% of cells, versus 84.1%.
- Code is published at github.com/MatrAIx-ai/PersonaCUABench.
- Products
- PersonaCUABench
- Organizations
- MatrAIx-ai
From the original article
1 min read at huggingface.co
Computer-use agents (CUAs) are commonly evaluated against fixed task specifications, but users may omit requirements, add constraints, or require approval before consequential actions. Completing the benchmark instruction does not guarantee serving the person who asked. We introduce PersonaCUABench, which holds 74 tasks across 11 domains fixed and varies only who asks.
By Haoming Li, Defu Cao, Xu Hu · huggingface.co
A short excerpt; the article and its images belong to the publisher. Summary, key facts and assessment above are AI-written from the article and can be wrong.