Jev 1.13 raises agent cost 19% without improving pass rate
Original title: Good Judgment, Bad Advice: Decision-Model Controllers Raise Agent Cost Without Improving Success
On 21 held-out Terminal-Bench 2.1 tasks, Jev matched a 73.0% pass rate but cost 22% more than a placebo.
A pre-specified study held a Claude Sonnet 5.5 terminal-agent loop fixed and varied only the controller: none, deterministic rules, Claude Haiku 5.5, Jev 1.13, or a placebo sampling Jev's development-phase choice frequencies. Across 315 trials, Jev was cheaper than the small-LLM controller (ratio 0.81), and no pass-rate difference between arms was detectable. Offline, Jev ranked finish requests better than alternatives but was poorly calibrated, so it requested verification of work that had already passed.
- Geometric-mean cost ratio versus no controller was 1.19, 95% CI 1.02-1.36.
- Finish-request AUROC was 0.78, versus 0.62 for rules and 0.60 for Haiku.
- Mean completion probability was 0.50 when 87% of those runs passed.
- No controller ever chose to abandon a run.
- Products
- Terminal-Bench 2.1
From the original article
1 min read at huggingface.co
Agent scaffolds increasingly delegate small decisions (whether to stop, verify, or change approach) to cheaper models. We ask whether a typed decision model, Jev 1. 13, consulted only at deterministic decision points, makes a Claude Sonnet 5.
By Naman Vats · huggingface.co
A short excerpt; the article and its images belong to the publisher. Summary, key facts and assessment above are AI-written from the article and can be wrong.