On the estimation and validity of AI time horizons---a statistical look at the METR plot
Paper recomputes METR AI time horizons with splines and item-response theory across 228 tasks.
The paper re-estimates METR's 50% time horizon, the human completion time of software tasks an AI solves with 50% probability. On 228 tasks and 26 AI systems, the authors use splines and item-response theory instead of assuming difficulty scales linearly with log human time. The fitted conversion is nearly flat from about 2 to 30 minutes and closer to linear elsewhere, so a jump from 3 minutes to 30 minutes is much easier than one from 30 minutes to 5 hours despite the same 10x multiplier. They report cross-validated point estimates and diagnostic plots for assessing construct validity.
- Reanalysis uses 228 tasks and 26 AI systems
- Fitted curve is nearly flat from about 2 to 30 minutes
- A 10x jump is much easier below 30 minutes than above
- Authors recommend diagnostic plots alongside time-horizon scores
From the original article
1 min read at arxiv.orgMETR's 50\% time horizon measures the human completion time of software tasks that an AI solves with 50\% probability, allowing AI capabilities to be expressed in interpretable units. On 228 tasks and 26 AIs, we recompute the time horizons using splines and item-response theory to relax the assumption that the AI difficulty of a task depends linearly on the log of human time.
By Drew T. Nguyen · arxiv.org
A short excerpt; the article and its images belong to the publisher. Summary, key facts and assessment above are AI-written from the article and can be wrong.