Harness-delta method finds much agent-harness gain is overfitting
Original title: Harness-Delta Attribution: Dissecting Gains From Harness Evolution
In 21 significant settings, overfitting and test-time scaling explain 54% and 14% of the gain on average.
Harness-Delta Attribution splits gains from optimizing prompts, tools, and orchestration around a fixed model into overfitting, test-time scaling, and a residual upper bound on generalizable improvement. The protocol covers 11 executor models and three harness-evolution frameworks on math, coding, creativity, and agentic tasks. Among 21 settings with significant search-set gains, the two non-generalizing factors exceed half the gain in 14. Search-set gains often fail to transfer, and validation signals do not prevent overfitting.
- Overfitting and test-time scaling average 54% and 14% of measured gain.
- Their combined share exceeds 50% in 14 of 21 settings.
- Residual transfer varies widely across tasks, models, and frameworks.
- Validation-based signals were insufficient to prevent overfitting.
From the original article
1 min read at huggingface.co
Harness evolution automatically optimizes the prompts, skills, tools, and orchestration code around a fixed-weight model, offering a way to improve agentic systems without updating model weights. The agent harness is iteratively optimized to achieve higher performance on the search set.
By Wenxuan Ding, Qin Lu, Changlong Yu · huggingface.co
A short excerpt; the article and its images belong to the publisher. Summary, key facts and assessment above are AI-written from the article and can be wrong.