FDE-Gym: The Last Mile of AI Delivery
FDE-Gym finds top agents resolve only 14 of 95 realistic customer-delivery tasks.
FDE-Gym is a benchmark of 95 end-to-end AI delivery tasks drawn from 20 customer scenarios across 18 sectors and adapted from real projects. Agents must clarify ambiguous requirements, interact with scripted stakeholders, and modify production-like systems; scores measure customer success uplift from 0 to 1 with interaction penalties. Across 13 configurations from seven providers, the best realistic-setting mean score was 0.56, with only 14 of 95 tasks resolved. Discovering unstated constraints was the main failure point, and full information still produced strong uplift on only 10 of 23 such tasks.
- FDE-Gym has 95 delivery tasks from 20 scenarios across 18 sectors.
- Best of 13 configurations scored 0.56 and fully resolved 14 tasks.
- Unstated constraints are the main bottleneck in realistic delivery.
- Even with all key facts, only 10 of 23 tasks reached 0.8 uplift.
From the original article
1 min read at arxiv.orgExisting agent benchmarks typically evaluate agents on well-specified tasks defined by issue reports, specifications, or task tickets. Real-world customer delivery is different: forward-deployed engineers (FDEs) must clarify ambiguous requirements, work in live systems, and uncover unstated constraints.
By Dongxu Yang · arxiv.org
A short excerpt; the article and its images belong to the publisher. Summary, key facts and assessment above are AI-written from the article and can be wrong.