ParetoSaddler improves quality and reduces cost in LLM agent harnesses
Original title: ParetoSaddler: Pareto-Guided Optimization of Agent Harnesses for Quality and Cost
ParetoSaddler uses Bayesian bootstrap Pareto support to promote candidates showing progress in quality or cost during optimization on ClawMark and GAIA2.
Harness optimization improves LLM agents without modifying weights. Existing methods focus on quality but struggle with joint quality-cost optimization. ParetoSaddler separates intermediate promotion from final selection. On ClawMark quality rises from 70.86% to 81.35% while cost drops from $1.17 to $1.01 per task. On GAIA2 quality improves from 56.22% to 62.89% and cost falls from $0.78 to $0.45 per task.
- ParetoSaddler retains intermediate candidates discarded by deployment preference
- Bayesian bootstrap filters weakly supported gains across runs
- Evaluated on domain-disjoint splits of ClawMark and GAIA2
- Pareto-selected harnesses achieve higher quality at lower cost than quality-only methods
From the original article
1 min read at huggingface.co
Harness optimization can substantially improve LLM agents without modifying model weights, but existing methods primarily optimize response quality. Extending them to joint quality--cost optimization introduces a challenge: applying the deployment preference during intermediate updates can prematurely reject candidates that are valuable for later improvements.
By Xingzhi Qian, Sungho Park, Rongyuan Tan · huggingface.co
A short excerpt; the article and its images belong to the publisher. Summary, key facts and assessment above are AI-written from the article and can be wrong.