Kolab benchmarks human-agent collaboration on long-horizon workflows
Original title: Kolab: Benchmarking Human-Agent Collaboration in Long-Horizon Workflows
The dataset covers 115-plus hours from 50 professionals in 40 occupations; frontier models overestimate collaboration quality.
Researchers introduce Kolab, a dataset of screen recordings of professionals working with AI agents on realistic long-horizon tasks, plus participants' judgments of collaboration and output quality. The benchmark asks models to forecast how experts adopt, modify, or reject agent contributions and to assess satisfaction and output quality without constant feedback. Frontier agentic and non-agentic models overestimate collaboration with experts and struggle to separate high- from low-quality collaboration. Longer workflow context does not consistently improve those predictions.
- Kolab includes more than 115 hours of recordings from 50 professionals.
- Participants come from 40 occupations and supply collaboration and output-quality judgments.
- Models must forecast adoption, modification, or rejection of agent contributions.
- Frontier models fail to distinguish high- and low-quality collaboration.
- Longer workflow context does not consistently improve predictions.
From the original article
1 min read at huggingface.co
When humans collaborate with AI agents on long-horizon tasks, what happens to the agents' contributions? For solving long-horizon tasks, humans place greater importance on ongoing interaction with AI agents (e. g.
By Shirley Anugrah Hayati, Patricia Anugrah Setiani, Ruizi Wang · huggingface.co
A short excerpt; the article and its images belong to the publisher. Summary, key facts and assessment above are AI-written from the article and can be wrong.