Google makes Gemini 4 Carbon generally available
Paid users can access Carbon now; Google says it complements Argon but spends more tokens on hard tasks.
Stories per day · 30 days
23 RSS
Relatedevaluation21llm-agents20agents16coding-agents11language-models11open-weights11llm-evaluation9multimodal9gemini8qwen8multimodal-agents7tool-use7google6robotics6artificial-analysis5kimi5self-improvement5agent-memory4
331 stories, newest first · one card per story
Stories tagged #benchmarks
Wiz relaunches Cyber Model Arena to score agent cyber tasks
Version 3.9 tests 59 agents on 327 challenges with a 1,000-second budget and deterministic grading.
Anthropic ships Claude Fable 5.2 and retakes the top LLM rank
Artificial Analysis ranks Fable 5.2 first at 65, ahead of Gemini 4 Carbon at 60.
1
Fable 5.2 completes 41.3 percent of freelance jobs in RLI test
Scale Labs says that is up from about 16 percent for Fable 5 in July and 2.5 percent for Manus a year ago.
Meta releases Muse Ember, a hosted 1M-context agent model
Weights stay private; Meta prices it at $0.10/$0.50 per million tokens and about $0.136 per AA task.
1
Veritas harness auto-generates agent-task verifiers that beat seven alternatives
With the model fixed, generated verifiers matched official verdicts better than seven harnesses on all four execution-scored benchmarks.
Claude Fable 5.2 scores 65 on Artificial Analysis, highest so far
Artificial Analysis Intelligence Index v5.0.1 rates Claude Fable 5.2 at 65, and the model has no public API.
MMDM benchmark tests 28 models on 5,448 decision tasks
Best hard multi-select accuracy is 36.81%; Decider-V1.2 falls from 78.75% on world knowledge to 47.95% on constraint planning.
PhyRubric scores physical commonsense errors in text-to-video models
A judge trained on 3,788 videos tracks human ratings more closely than other open judges and matches Gemini 4 Argon.
Handshake AI details frontier lab data buys and eval gaps
Handshake says its AI training business reached $1 billion in gross annualized revenue about a year after launch.
1
Baidu says ERNIE Evolve beats NanoChat and NanoGPT records
The retrieved page is a cookie notice, so scores, methods and release status are not stated.
EmoHarness raises small multimodal emotion scores with audio evidence
On 5,507 MME-Emotion items, Recognition rose 7.6-13.4 points, mostly from transcripts and prosody rather than evolved edits.
Embodied First Exam tests coding agents on 1,156 robot tasks
GPT-6 Astra in Codex, strongest of eight agents, succeeds on 34% of 456 core tasks.
Spatial Harness grounds VLM spatial reasoning in physics-engine states
On MindCube it raises Qwen3.5-27B from 52.70% to 71.14%, and to 98.25% when poses are annotated.
WorldFoundry catalogs 270 models for world-model evaluation
The open-source stack also lists 74 benchmarks and 62 metrics, separating execution evidence from scores.
Longtail-SWE adds 202 SWE tasks for 13 underrepresented languages
A solver-judge-repairer pipeline revises tests and descriptions; nine model setups were evaluated and artifacts released.
Tencent releases Hy4 with vision and computer use; weights come later
Hy4 is API-only for now, scores 35 on Artificial Analysis, and trails Kimi K3.1 Pro by a wide margin.
Siris tests whether LLMs spot invalid simulator recommendations
On LLM serving design problems, spotting a simulator limitation did not guarantee the right design, and a review turn added new errors.
ReproBridge lifts ML reproduction success from 76.4% to 92.6%
Across 1,029 tasks from 387 experiments, refined artifacts also cut mean API spending by 19.7%.
No AI agent scores above 12 percent on SemiAnalysis GDPx
Opus 5.5 leads across 206 simulated tasks; 98 percent of cyber runs broke a behavioral rule.
1
FINCOSMOS evaluates finance agents without fixed tasks, data, or timelines
In 150 sessions no run passed; agents surfaced only 54-64% of untold details, and experts blamed them for 88.33% of failures.
Text-to-Image Models Fail to Count Objects Past Ten
Five open models follow number words only up to ten objects; SD3.5-M and FLUX.1-dev ignore counts beyond that.
LLMs Apply Time-Dependent Constraints in Only 30.5% of Cases
Thirteen models correctly calculate elapsed time in 62.1% of requests but fail to apply constraints.
VisSearch-Bench benchmarks visual search for multimodal agents
Best model solves only 26.47% of 204 tasks, showing challenges in linking visual evidence.