Paper finds most long-horizon benchmarks lack depth
Original title: Is Your Long-Horizon Benchmark Really Deep? Measuring and Synthesizing Depth in Terminal Tasks
Audit reveals 94% of synthetic tasks pass in three turns yet depth predicts difficulty better; authors propose DEPTH method and release 105 harder tasks.
Long-horizon terminal tasks evaluate agents but human datasets are scarce, so synthesis methods have emerged. Turn count largely reflects model habits and labels barely track passing rates. The authors define depth as the smallest turn cap under which a task passes and audit 734 tasks from six datasets. Their DEPTH generation method produces 105 tasks exceeding existing ones in depth and work; none of six models pass half. Fine-tuning Qwen3-8B on these runs improves it on two terminal benchmarks.
- 94% of synthetic tasks pass within three turns
- Depth predicts difficulty beyond work
- 105 new DEPTH tasks exceed audited datasets
- None of six models pass half of new tasks
- Fine-tuning Qwen3-8B boosts benchmark performance
From the original article
1 min read at huggingface.co
Long-horizon terminal tasks are used to evaluate and train agents. Human-curated datasets are scarce, so many synthesis methods have emerged. They usually measure the length of a task by the turns a model needs to pass it, and its difficulty by labels the LLM sets during generation.
By Yu Jiang, Yue Huang, Hang Hua · huggingface.co
A short excerpt; the article and its images belong to the publisher. Summary, key facts and assessment above are AI-written from the article and can be wrong.