REALCLI scores frontier CLI agents poorly on collaboration
Original title: RealCLI: Toward a Unified Evaluation of Human-Agent Collaboration for CLI Agents
The best configuration averaged 49.2%, and the top joint collaboration score was 37.6%.
REALCLI is a paired benchmark for command-line agents that scores both task execution and human-agent collaboration. Collaboration is defined with five dimensions, 22 sub-dimensions, and 51 interaction scenario templates. REALCLI-EXEC gives full specifications up front, while REALCLI-COLLAB uses a scripted user that adds information, corrections, constraint changes, and authorization decisions over time. Reported results show large gaps: models struggle with steerability, trust calibration, and safety-boundary management, and task success diverges from collaboration quality.
- The framework has five dimensions, 22 sub-dimensions, and 51 scenario templates.
- REALCLI-EXEC and REALCLI-COLLAB support paired comparison.
- Peak joint collaboration score J_collab was 37.6%.
- Weak spots named are steerability, trust calibration, and safety boundaries.
From the original article
1 min read at huggingface.co
Command-line interface (CLI) agents increasingly execute longer-horizon tasks, yet existing benchmarks either assume that user instructions remain fixed throughout execution or evaluate task outcomes without capturing runtime collaboration quality. In this paper, we present REALCLI, a paired benchmark that unifies the evaluation of task execution and human-agent collaboration quality on realistic CLI tasks.
By Yunqi Liu, Peiyu Wang, Bingli Wang · huggingface.co
A short excerpt; the article and its images belong to the publisher. Summary, key facts and assessment above are AI-written from the article and can be wrong.