HERA co-evolves agent harnesses to abstain on infeasible tasks
Original title: HERA: Harness-Environment Co-Evolution for Reliable Agentic Abstention
An evolved harness raised abstention accuracy from 61.7% to 83.3% and transferred to 19 other LLMs.
HERA is a framework for harness-environment co-evolution aimed at agentic abstention, when an LLM agent should recognize that a tool-use task has no valid solution. It builds verifiable feasible and infeasible task pairs by mutating environments, then uses prior failures to adapt the harness and generate harder tasks. On held-out tasks the evolved harness also raised feasible-task completion from 68.3% to 76.7%. The best harness transferred across 19 other LLMs without model-specific optimization.
Unconfirmed: Reported accuracy, transfer, and cost figures are the authors' own evaluation results.
- Abstention accuracy rose from 61.7% to 83.3% on held-out tasks.
- Feasible-task completion rose from 68.3% to 76.7%.
- The harness improved abstention by 15.3 points on average across 19 LLMs.
- Authors estimate smaller models can match stronger ones at about 85% lower cost.
- Mutations turn solvable tasks into cases that require abstention.
From the original article
1 min read at huggingface.co
Large language model (LLM) agents are increasingly capable of acting in complex tool-use environments, yet they often fail to recognize when tasks are infeasible and no valid solution exists.
By Han Luo, Bingbing Wen, Guang Yang · huggingface.co
A short excerpt; the article and its images belong to the publisher. Summary, key facts and assessment above are AI-written from the article and can be wrong.