DAYJOB: A Benchmark for Long-Horizon Professional Work
DAYJOB finds even Claude Opus 5.5 passes under 25% of long professional healthcare and finance tasks.
DAYJOB is a benchmark of 130 long-horizon professional tasks, 50 in healthcare and 80 in finance, built as containerized Harbor environments with expert binary rubrics. Tasks are estimated at 13.6 hours on average in healthcare and 16.6 in finance, and an attempt passes only if it meets every criterion (median 47.5 and 57.5). Across 30 model configurations from 13 developers, Claude Opus 5.5 is strongest at 24.7% healthcare and 23.9% finance pass rates, while the median configuration scores 0.6% and 2.5%. Case studies show agents accepting premises the record contradicts. Healthcare tasks, 50 of 80 finance tasks, the harness, and a leaderboard are released.
- 130 professional tasks: 50 healthcare, 80 finance.
- Healthcare tasks average 13.6 hours; finance 16.6 hours.
- Pass requires every binary rubric criterion; medians are 47.5 and 57.5.
- Claude Opus 5.5 passes 24.7% healthcare and 23.9% finance; median config under 3%.
- Agents accept contradicted premises and propagate wrong inputs.
From the original article
1 min read at arxiv.orgProfessional work often starts with a brief request that leaves the professional to work out what is needed, which documents matter, and whether the request's premise holds. We introduce DAYJOB, a benchmark of 130 tasks built by professionals in healthcare (50) and finance (80). The tasks are estimated to take a professional 13.
By Stephanie Finley · arxiv.org
A short excerpt; the article and its images belong to the publisher. Summary, key facts and assessment above are AI-written from the article and can be wrong.