EVOHARNESSBENCH: Can Your Agents Keep Pace with an Evolving Harness?
Researchers introduce EVOHARNESSBENCH, a benchmark showing that evolving agent harnesses (tools, skills, agents) cause forgetting and inconsistent adaptation across 802 tasks.
The paper introduces EVOHARNESSBENCH, a benchmark that places non-stationarity in the externally supplied agent harness rather than in the task stream, evaluating agents across tools, skills, and specialist agents. It comprises 17 multi-stage harness streams built deterministically from verifier-based benchmarks, totaling 802 tasks, 520 tools, 42 skills, and 62 agents. Evaluation covers deployment (retention of previously accessible competence) and self-evolving adaptation settings. Results show harness expansion alone degrades previously solved tasks (harness-induced forgetting), adaptation gains are inconsistent, and retention and adaptation can pull in opposite directions.
- Benchmark puts non-stationarity in the harness itself (tools, skills, agents), unlike task-stream continual-learning benchmarks.
- 17 deterministically constructed harness streams: 802 tasks, 520 tools, 42 skills, 62 agents.
- Harness expansion alone degrades performance on previously solved tasks, termed harness-induced forgetting.
- Retention and adaptation can conflict; preserving earlier competence does not guarantee adapting to new capabilities.
From the original article
1 min read at huggingface.co
Modern LLM-based agents operate through a harness of tools, reusable skills, and specialist agents that shapes what they observe and what they can do. In practice, this harness continually evolves as new capabilities are added. We introduce EVOHARNESSBENCH, a benchmark for evaluating agents under controlled harness evolution across three axes (tools, skills, and agents).
By Zixuan Ke, Vaidehi Patil, Haizhou Shi · huggingface.co
A short excerpt; the article and its images belong to the publisher. Summary, key facts and assessment above are AI-written from the article and can be wrong.