Higher LLM reasoning effort can cost more without better results
Original title: What Effort Level Should I Choose? On the Economics of Reasoning Effort in LLM Agents
On easy tasks every effort level solves, the highest setting costs up to 42 times the lowest, and escalating after failure is often cheaper.
The paper treats LLM reasoning-effort settings as a credence good and measures their counterfactual effect on proprietary and open-weight models across four benchmarks, covering single-call reasoning and long-horizon agent tasks. Higher effort can raise aggregate accuracy but often fails tasks that lower effort already solved, and it changes how agents work by increasing self-verification and search. On easy tasks solved at every level, higher effort is generally more expensive. Starting at the lowest effort and escalating only after a verified failure matches or improves on fixed top effort across all tested models and benchmarks, at lower cost for most proprietary models.
Unconfirmed: The paper does not name the four benchmarks or the specific proprietary and open-weight models in this abstract.
- The highest effort level costs up to 42 times the lowest on those easy tasks.
- Higher effort makes agents search more often instead of working problems directly.
- Longer trajectories do not necessarily change the outcome.
- The escalate-after-failure policy lowers cost for most proprietary models tested.
From the original article
1 min read at huggingface.co
Generative models increasingly expose a reasoning effort setting that lets users scale inference-time compute, trading higher cost for potentially improved performance. Users must choose an effort level before knowing whether extra reasoning will help, and even a successful run does not reveal whether the effort purchased was necessary.
By Kaiser Sun, Yongchan Kwon, Zain Hasan · huggingface.co
A short excerpt; the article and its images belong to the publisher. Summary, key facts and assessment above are AI-written from the article and can be wrong.