How Much of a Harness Does a Strong Agent Need for Autonomous ML Engineering?
Ablations show state-of-the-art MLE-agent harnesses offer no advantage over one minimal coding-agent session when the frontier LLM backbone and time budget are equal.
The paper studies autonomous ML engineering agents deployed on increasingly elaborate harnesses such as multi-agent orchestrators and dedicated retrieval subagents. Under equal time budgets and the same frontier LLM backbone, open-source state-of-the-art harnesses provided no advantage over a single session of a minimal-harness coding agent with direct read, write, and bash access. Large-scale ablations indicate the machinery layers are redundant in the coding agent setting, with the backbone the primary performance driver.
- Minimal coding agent matches SOTA MLE harnesses under equal time budget.
- Backbone LLM quality is the primary driver of MLE benchmark performance.
- Multi-agent orchestrators and retrieval subagents become redundant layers.
- Hand-crafted harnesses yield poor returns on current MLE benchmarks.
From the original article
1 min read at arxiv.orgRecent autonomous machine learning engineering (MLE) agents have made significant progress on public leaderboards. Often motivated by progress stagnation over long-horizon cycles and limited Large Language Model (LLM) primitives, modern MLE agents are deployed on top of increasingly elaborate machinery: multi-agent orchestrators, dedicated retrieval subagents, and more.
By Kirill Brilliantov · arxiv.org
A short excerpt; the article and its images belong to the publisher. Summary, key facts and assessment above are AI-written from the article and can be wrong.