AHE-Beta evolves agent harnesses from user requirements
Original title: Agentic Harness Engineering Beta: Requirement-Centered Evolution of Agent Harnesses
On held-out tasks, pass@1 rose from 21.9% to 33.3% while using 40.8% fewer tokens than Base Harness.
The paper introduces Agentic Harness Engineering Beta, a loop that collects simulated-user requirements, implements them as generalizable harness changes, and merges a change only after it resolves a verified requirement and passes review. On held-out tasks it reports higher pass@1 than Base Harness, Claude Code, Meta-Harness, and AHE, with lower token use than Base Harness. In a blind human test, users preferred AHE-Beta in 31 of 48 valid pairs. Without further evolution, gains transferred to GLM-5.3 and DeepSeek-V4.1-Flash.
Unconfirmed: Reported gains are the authors' benchmark and blind-test results, not an independent evaluation.
- Pass@1 rose from 21.9% to 33.3% versus Base Harness.
- Token use was 40.8% lower than Base Harness.
- Users preferred AHE-Beta in 31 of 48 valid pairs.
- Transfer gains were 15.8% on GLM-5.3 and 5.3% on DeepSeek-V4.1-Flash.
- Products
- Claude CodeMeta-HarnessAHE-Beta
- AI models
- GLM-5.3DeepSeek-V4.1-Flash
From the original article
1 min read at arxiv.orgReal-world harness engineering is driven by explicit and implicit user requirements, which guide development and validation. However, existing harness evolution methods often overlook implicit requirements and interaction needs, while using a single evolution agent to modify the harness and benchmark scores to select changes. This process often produces narrow patches and leads to benchmark-specific local optima.
By Jiahang Lin · arxiv.org
A short excerpt; the article and its images belong to the publisher. Summary, key facts and assessment above are AI-written from the article and can be wrong.