Can LLMs Engineer Their Own Agent Harness? ByteDance Seed’s HarnessDev Says Only 34 of 64 Changes Generalize
ByteDance Seed's HarnessDev benchmark finds LLM-built agent harnesses trail human engineering on code and search, with only 34 of 64 revisions generalizing.
Researchers from ByteDance Seed, SUTD, Georgia Tech, M-A-P, and TokenWave.AI introduce HarnessDev, a benchmark that evaluates the runnable agent harness an LLM writes rather than its answers, using Creation and Evolution stages across SWE-bench Pro, Terminal-Bench 2.1, MLE-bench, EQ-Bench3, and BrowseComp (2,207 instances). Six creator models including Opus 4.8, GPT-5.5, Gemini 3.1 Pro, DeepSeek V4 Pro, Qwen 3.7 Max, and Seed 2.0 Pro were tested; Opus 4.8 posted the best average of 67.8 versus an 86.2 human-engineered reference. Self-built harnesses beat references on writing and ML experimentation but lag badly on code and search, and quality proved executor-specific: Opus 4.8 fell from 69.3 to 33.0 on SWE-bench Pro when the executor was switched to Gemini. Evolution gains were small and noisy: of 64 adjacent changes, feedback and held-out scores agreed only 34 times (53.1%), and much generated state and memory code never executed.
- HarnessDev grades the runnable harness a model writes, not its answers, spanning code, search, writing, and ML-experimentation benchmarks.
- Opus 4.8 averaged 67.8 versus an 86.2 human reference; writing and MLE-bench results beat references, search trails most.
- Harness quality is executor-specific: Opus 4.8 dropped from 69.3 to 33.0 on SWE-bench Pro under a Gemini executor.
- Only 34 of 64 evolution changes aligned with held-out gains; 2 of 9 declared final versions were held-out optimal.
- Much generated machinery is inert: 18 of 108 code components never fired and 124 of 587 writing features were dead code.
From the original article
4 min read at marktechpost.com
ByteDance Seed, SUTD, Georgia Tech, M-A-P, and TokenWave. AI introduce HarnessDev, a benchmark that scores the runnable harness a model builds rather than the answer it returns. Starting from a seed that scores 0, 6 creator LLMs construct harnesses across 5 benchmarks and 2,207 tasks, then evolve them from execution feedback.
By Asif Razzaq · marktechpost.com
A short excerpt; the article and its images belong to the publisher. Summary, key facts and assessment above are AI-written from the article and can be wrong.