Can LLMs Engineer Their Own Agent Harness? ByteDance Seed’s HarnessDev Says Only 34 of 64 Changes Generalize
ByteDance Seed's HarnessDev benchmark finds LLM-built agent harnesses trail human engineering on code and search, with only 34 of 64 revisions generalizing.
Researchers from ByteDance Seed, SUTD, Georgia Tech, M-A-P, and TokenWave.AI introduce HarnessDev, a benchmark that evaluates the runnable agent harness an LLM writes rather than its answers, using Creation and Evolution stages across SWE-bench Pro, Terminal-Bench 2.1, MLE-bench, EQ-Bench3, and BrowseComp (2,207 instances). Six creator models including Opus 4.8, GPT-5.5, Gemini 3.1 Pro, DeepSeek V4 Pro, Qwen 3.7 Max, and Seed 2.0 Pro were tested; Opus 4.8 posted the best average of 67.8 versus an 86.2 human-engineered reference. Self-built harnesses beat references on writing and ML experimentation but lag badly on code and search, and quality proved executor-specific: Opus 4.8 fell from 69.3 to 33.0 on SWE-bench Pro when the executor was switched to Gemini. Evolution gains were small and noisy: of 64 adjacent changes, feedback and held-out scores agreed only 34 times (53.1%), and much generated state and memory code never executed.