Δτ-Bench tests agents as tool ecosystems change
Original title: Δτ-Bench: Pushing Agents Toward a Moving Capability
The strongest evaluated model or harness reached only 42.3% success on 1,588 tasks across six ecosystem states.
Δτ-Bench is an executable agent benchmark in which independently evolving services remove, replace, or add tools while documentation stays noisy. It contains 1,588 validated multi-turn tasks across 42 domains and six successive ecosystem states produced by capability-preserving transformations. Even the strongest evaluated model or harness reached only 42.3% task success, and existing skill-learning gains did not consistently improve adaptation as the ecosystem changed. The authors also introduce Steward, which maintains tools and procedural knowledge through incremental skill maintenance and dynamic provisioning.
Unconfirmed: The paper does not name which model or harness achieved the 42.3% figure.
- The suite has 1,588 multi-turn tasks, 42 domains, and six ecosystem states.
- Top evaluated success was 42.3%.
- Skill-learning gains did not consistently carry over as tools evolved.
- Steward jointly manages tools and procedural knowledge during execution.
- Experiments report higher success and lower skill-maintenance overhead with Steward.
From the original article
1 min read at huggingface.co
In this paper, we introduce Δτ-Bench, an executable agent benchmark that models the structural evolution of large tool ecosystems. In Δτ-Bench, independently evolving services continually reshape how tasks can be accomplished: familiar tools may disappear or be replaced, while more efficient alternatives emerge amid overlapping functionality and noisy documentation.
By Shuofei Qiao, Xuanguang Pan, Yijun Chen · huggingface.co
A short excerpt; the article and its images belong to the publisher. Summary, key facts and assessment above are AI-written from the article and can be wrong.