EVOHARNESSBENCH: Can Your Agents Keep Pace with an Evolving Harness?
Researchers introduce EVOHARNESSBENCH, a benchmark showing that evolving agent harnesses (tools, skills, agents) cause forgetting and inconsistent adaptation across 802 tasks.
The paper introduces EVOHARNESSBENCH, a benchmark that places non-stationarity in the externally supplied agent harness rather than in the task stream, evaluating agents across tools, skills, and specialist agents. It comprises 17 multi-stage harness streams built deterministically from verifier-based benchmarks, totaling 802 tasks, 520 tools, 42 skills, and 62 agents. Evaluation covers deployment (retention of previously accessible competence) and self-evolving adaptation settings. Results show harness expansion alone degrades previously solved tasks (harness-induced forgetting), adaptation gains are inconsistent, and retention and adaptation can pull in opposite directions.
- Benchmark puts non-stationarity in the harness itself (tools, skills, agents), unlike task-stream continual-learning benchmarks.
- 17 deterministically constructed harness streams: 802 tasks, 520 tools, 42 skills, 62 agents.
- Harness expansion alone degrades performance on previously solved tasks, termed harness-induced forgetting.
- Retention and adaptation can conflict; preserving earlier competence does not guarantee adapting to new capabilities.
Full article235 words · extracted from huggingface.co · click to collapse
Modern LLM-based agents operate through a harness of tools, reusable skills, and specialist agents that shapes what they observe and what they can do. In practice, this harness continually evolves as new capabilities are added. We introduce EVOHARNESSBENCH, a benchmark for evaluating agents under controlled harness evolution across three axes (tools, skills, and agents). Unlike existing continual-learning benchmarks for agents, which typically place non-stationarity (i.e., what changes over time) in the task stream while keeping the harness fixed, EVOHARNESSBENCH places non-stationarity in the externally supplied harness itself. It contains 17 multi-stage harness streams constructed deterministically from verifier-based benchmarks, comprising 802 tasks, 520 tools, 42 skills, and 62 agents. We evaluate two complementary settings corresponding to the central challenges of harness evolution: deployment evaluation, which isolates retention of previously accessible competence as the harness expands, and self-evolving adaptation evaluation, which tests whether accumulated experience remains useful as new capabilities are introduced. Our results reveal three persistent gaps. First, harness expansion alone can degrade performance on previously solved tasks, producing harness-induced forgetting. Second, gains from self-evolving adaptation remain inconsistent across stages of harness evolution, capability axes, and environments. Third, retention and adaptation can pull in different directions: preserving earlier competence does not necessarily improve adaptation to newly introduced capabilities, and vice versa. These results establish harness evolution as a distinct challenge for building agents that can keep pace with an evolving harness while preserving previously effective behavior.
Text extracted automatically; images, tables and formatting may be missing. Original: https://huggingface.co/papers/2609.04280