arXiv study finds synthetic AI tasks lack true depth
94% of synthetic tasks pass within three turns but depth predicts difficulty better than turn count.
Stories per day · 30 days
2 RSS
Relatedreinforcement-learning2agentic-evaluation1dashboard1depth1environment-generation1eu-ai-act1evaluation1globebench1model-evaluation1responsible-ai1terminal-tasks1world-models1
7 stories, newest first · one card per story
Stories tagged #llm-benchmarks
VERA Dashboard Scores LLM Deployments for EU AI Act
VERA dashboard reduces responsible AI assessment time from 56 to 30 seconds on average while scoring twelve measurable requirements.
Environment Duels benchmark scores LLMs as task authors and solvers
Across nine frontier models, the best solvers are not the best authors, and smaller models write the hardest environments.
GlobeBench measures LLM text world-model simulation accuracy
Across 1,632 tasks, GPT-6 Astra leads 20 models at 49.57%, and 631 tasks stay unsolved.
Can LLMs Engineer Their Own Agent Harness? ByteDance Seed’s HarnessDev Says Only 34 of 64 Changes Generalize
ByteDance Seed's HarnessDev benchmark finds LLM-built agent harnesses trail human engineering on code and search, with only 34 of 64 revisions generalizing.
9
A Misalignment of AI in Mathematics
25 Fields Medallists including Terence Tao issue a declaration warning that AI companies' benchmark-driven mathematics goals are misaligned with science and society.
6
Benchmark Scores Are Pipeline-Dependent: A Reliability Audit of Cybersecurity LLM Benchmarks
Audit of eight cybersecurity LLM benchmarks shows evaluation pipeline choices can swing scores by over 80 points and reshuffle most model rankings.