EVOHARNESSBENCH: Can Your Agents Keep Pace with an Evolving Harness?
Researchers introduce EVOHARNESSBENCH, a benchmark showing that evolving agent harnesses (tools, skills, agents) cause forgetting and inconsistent adaptation across 802 tasks.
The paper introduces EVOHARNESSBENCH, a benchmark that places non-stationarity in the externally supplied agent harness rather than in the task stream, evaluating agents across tools, skills, and specialist agents. It comprises 17 multi-stage harness streams built deterministically from verifier-based benchmarks, totaling 802 tasks, 520 tools, 42 skills, and 62 agents. Evaluation covers deployment (retention of previously accessible competence) and self-evolving adaptation settings. Results show harness expansion alone degrades previously solved tasks (harness-induced forgetting), adaptation gains are inconsistent, and retention and adaptation can pull in opposite directions.