The Tasteful Agent: Measuring and Improving Taste in Long-Horizon Tasks
Researchers introduce Taste-Bench, a benchmark measuring LLM agents' ability to make good long-horizon decisions at critical decision forks.
This paper introduces Taste-Bench, a benchmark designed to measure the 'taste' of LLM agents, defined as their ability to make good long-horizon decisions at critical forks. The benchmark is automatically constructed from agent trajectories in engineering and research tasks, presenting decision points where one direction leads to a better outcome. Evaluation shows that even the best frontier model answers only 59.7% of taste questions correctly. The researchers also demonstrate that taste can be improved by distilling judgment from a teacher model that has access to the full trajectory outcome.
- Introduces Taste-Bench to measure 'taste' in LLM agents for long-horizon tasks.
- Taste is defined as the ability to make good long-horizon decisions at decision forks.
- The best model achieves only 59.7% accuracy on the benchmark.
- Taste can be improved by distilling judgment from a teacher model that sees outcomes.
Full article264 words · extracted from huggingface.co · click to collapse
LLM agents increasingly work on long-horizon tasks, and the decisions they make along the way, such as which hypothesis to test or which implementation to build on, determine the outcome of the whole run. Making these decisions well is becoming a key capability for both engineering and research agents. We refer to the ability to make good long-horizon decisions as the taste of an agent. While existing benchmarks measure the end-to-end success of agents on long-horizon tasks, none of them measures the taste of an agent. To address this problem, we build Taste-Bench, a benchmark of taste questions constructed automatically from trajectories that agents produced in engineering and research tasks. Each question presents a decision fork, a point in a trajectory where multiple directions are available and one of them leads to a better outcome, and the evaluated model chooses among these directions without seeing what happens after the fork. We mine these forks automatically from parallel attempts at the same task and from detours inside a single trajectory, without needing human annotation. We evaluate frontier models on Taste-Bench and find that the best model answers only 59.7% of the questions correctly. We further find that forks whose deciding evidence appears later in the trajectory are much harder for every model, and that a larger reasoning budget does not improve the accuracy. Finally, we show that taste can be trained. We distill the judgment of a teacher that has seen the outcome into a student model, and the student makes better decisions on unseen tasks and improves end-to-end success on held-out SWE-bench Pro tasks.
Text extracted automatically; images, tables and formatting may be missing. Original: https://huggingface.co/papers/2609.25804