Skip to content
NewA calmer, faster ZeroHour

METR update: models still lag humans on judgment tasks

Original title: The most misunderstood graph in AI just got a major update

MIT Technology Review · AIby Grace Huckins ()6 min read
AI summary · x-ai/grok-4.7Summaries by Vani AI

On the original plot, completable task length doubled about every four months, but no model produced an evaluation METR staff would use.

Nonprofit METR updated its time-horizon plot with longer and more open-ended tasks. Models remain stronger on automatically checkable work such as coding and math than on judgment-heavy work. In one task, models designed AI evaluations and METR employees scored them; none was useful for research. Models also fell short of humans when scoring philosophical arguments. Some newer models, including Opus 5.5, reached borderline performance on evaluation design, up from clearly below the bar a year earlier.

Unconfirmed: METR staff disagree on how relevant the philosophy task is to automating AI R&D, and the report includes few judgment tasks, so extrapolation is uncertain.

Key facts
  • METR scored task difficulty by how long human testers took to finish.
  • Not one AI-designed evaluation was good enough for METR research use.
  • Opus 5.5 was among models METR called borderline above the bar.
  • A year earlier every model scored clearly below the bar on those tasks.
  • OpenAI said its models do worse on employees’ real tasks than on METR’s original suite.
Explain this AI news with Vani AIWhat was announced, what is new and why it matters, in a new chat

From the original article

6 min read at technologyreview.com

A new report from the AI evaluation nonprofit METR finds that some AI capabilities are continuing to improve along an exponential trajectory—but it also indicates that models aren’t quite as good at tasks that require judgment as they are at coding. The report is an update to the organization’s time horizon plot, which was widely…

MIT Technology Review · AI

By Grace Huckins · technologyreview.com

Read the full article at MIT Technology Review · AI

A short excerpt; the article and its images belong to the publisher. Summary, key facts and assessment above are AI-written from the article and can be wrong.