FDE-Gym: The Last Mile of AI Delivery
FDE-Gym shows frontier agents still fail most real customer-delivery tasks, fully resolving only 14 of 95.
FDE-Gym is a benchmark of 95 end-to-end AI delivery tasks drawn from 20 customer scenarios across 18 sectors and adapted by industry experts from real projects. Agents must clarify ambiguous requirements with scripted stakeholders and modify production-like systems; scores reflect improvement in the customer’s success metric, normalized from 0 to 1, with penalties for messages, trial runs, and waiting. Across 13 configurations from seven providers, the best realistic-setting result was a mean score of 0.56, fully resolving 14 of 95 tasks. Even when every key fact was obtained on 23 tasks, the best configuration reached a business uplift of at least 0.8 on only 10.
- FDE-Gym contains 95 delivery tasks from 20 scenarios across 18 sectors.
- Scores measure customer-metric improvement from 0 to 1, with interaction penalties.
- The best of 13 configurations scored 0.56 and fully resolved 14 of 95 tasks.
- Finding unstated constraints is the main bottleneck for current agents.
From the original article
1 min read at huggingface.co
Existing agent benchmarks typically evaluate agents on well-specified tasks defined by issue reports, specifications, or task tickets. Real-world customer delivery is different: forward-deployed engineers (FDEs) must clarify ambiguous requirements, work in live systems, and uncover unstated constraints.
By Dongxu Yang, Hongda Yan, Renxin Wang · huggingface.co
A short excerpt; the article and its images belong to the publisher. Summary, key facts and assessment above are AI-written from the article and can be wrong.