arXiv study finds synthetic AI tasks lack true depth
Original title: Is Your Long-Horizon Benchmark Really Deep? Measuring and Synthesizing Depth in Terminal Tasks
94% of synthetic tasks pass within three turns but depth predicts difficulty better than turn count.
Long-horizon terminal tasks evaluate agents but human-curated datasets are scarce. Synthesis methods measure length by model turns and difficulty by LLM labels during generation. Audit of 734 tasks from six datasets shows synthesis mostly adds work not depth. The proposed DEPTH method where agents write tasks segment by segment in real environments creates 105 harder tasks none of six models pass more than half of. Fine-tuning Qwen3-8B on these improves performance on two terminal benchmarks.
- 94% of synthetic tasks pass in three turns
- depth unlike turn count predicts difficulty
- 105 new tasks exceed existing datasets
- no current model passes half of them
- fine-tuning improves two benchmarks
- AI models
- qwen3-8b
From the original article
1 min read at arxiv.orgLong-horizon terminal tasks are used to evaluate and train agents. Human-curated datasets are scarce, so many synthesis methods have emerged. They usually measure the length of a task by the turns a model needs to pass it, and its difficulty by labels the LLM sets during generation.
By Yu Jiang · arxiv.org
A short excerpt; the article and its images belong to the publisher. Summary, key facts and assessment above are AI-written from the article and can be wrong.