Pyrite benchmarks LLM agents on protein-ligand pose ranking
On 1,500 pose pairs, PyriteFeaturizer reaches 74% accuracy and PyriteReasoner 67%, while expert chemists perform near chance.
Stories per day · 30 days
44 RSS
Relatedllm-agents27multimodal19robotics12agents11evaluation9ai-agents8multi-agent8reinforcement-learning8llm-evaluation7huggingface6mllm6qwen6coding-agents5dataset5embodied-ai5image-generation5llm5prompt-injection5
595 stories, newest first · one card per story
Stories tagged #benchmark
Continual-ARC scores lifelong skill acquisition on ARC-AGI streams
Harness persistence rarely beats a demonstration cache on memorization, and no harness method gains on unseen tasks, unlike weight updates.
Re-Search tests whether LLMs can rediscover published research ideas
On 200 ICML 2026 papers, six LLMs make nearly all progress in the first 30 of 100 yes-or-no questions.
LLM world models pass mechanical checks but miss intended structure
Of 134 valid game-track models, mean structural F1 is 0.366 and only 28 exactly match the reference.
Audit says CAGE 4 defender scores measure less than assumed
Busy-agent handling, unseeded policies, and unpublished scoring choices make published CAGE Challenge 4 comparisons hard to reproduce.
BypassGym tests whether AI agents can evade deployed EDR
DeepSeek-V4.1 Flash succeeded on 17 of 18 runs; Bitdefender blocked all but 5 of 15.
WorkflowArena benchmarks runtime workflow orchestration by LLM agents
On a hard subset, Claude Opus 5.5-authored workflows substantially improved weaker executors with the executor held fixed.
Benchmark plants 100 errors in psychology papers to test AI review
The best system found 71 errors and the pooled systems found 93; seven missed errors were all omissions.
Sage-Taiyi leads Taiyi-Bench scientific ideation scores
It scores 54.0 overall versus 47.9 for AI Scientist-v2, and authors report chemistry yield gains in collaborations.
ShadowBench-MCP measures shadow invocations in MCP-style tool routing
On a Groq-hosted qwen3-32b router, efficiency-first routing had a 29.2% shadow-invocation rate versus 4.2% when policy-aware.
Infrathon tests language agents on long-horizon AI infrastructure tasks
Across 150 runs and over 10,000 GPU-hours, 135 still improved in the second half of a 24-hour budget.
THEATRON benchmark tests persona persistence in long LLM decisions
Four LLMs show persona-driven shifts that often fail to become stable, context-sensitive policies over multi-month financial trajectories.
GOPR-CAD builds CAD models through progressive operation graphs
On a 300-part MHCAD benchmark, the training-free pipeline reaches 82.7% Structural Yield.
ArmedAgent finds safety risks in coding agents that control robot arms
The benchmark covers 36 tasks in five scenarios, including indirect prompt injection that can move from digital resources into physical actions.
SurgKnowBench and SurgAgentGym test generalist models on surgery scenes
Agentic tool use mainly helps perception-heavy tasks; localization and expert-style assessment stay weak.
AI coding agents still need human cleanup on nearly three out of four tasks in a real-world test
Co-Design Bench shows top models like GPT-6 Astra complete only 26% of real tasks, requiring human cleanup.
1
ErosionBench scores alignment decay in enterprise multi-agent systems
The benchmark has 1,728 scenarios and 198 image attacks; a Meta-Auditor reached 98.5% recall at 2.9% false positives.
MaxxBench exposes DeepSWE overfitting in AI coding models
Mutations to DeepSWE tasks cause large performance drops in models due to benchmark overfitting.
No AI agent met the expert bar on open national-lab research tasks
Of 135 submissions from nine agents, 114 failed integrity checks, and only GPT-6 Astra produced one complete paper.
1
MotionGraphicsBench scores SVG motion edits with verification programs
Pass@1 across 47 configurations ranged from 6.0% to 67.7%, and verifiers agreed with humans 93.3% of the time.
Authorization drift occurs in LLM agents during multi-step tool use
Benchmark shows frontier LLMs drift from authorization boundaries in 43% to 61% of tasks, but clean-success paths exist for 86% of cases.
EarthRecord tests agents on four decades of raw satellite imagery
Twelve agents mostly finish analyses, but half of correct answers follow an altered satellite when the ground is unchanged.
Frontier3DBench benchmarks coding agents for 3D creation
GPT-6.1-sol leads object and scene levels on Frontier3DBench while reference reconstruction remains a key bottleneck
XStack benchmark evaluates ML repository migration between PyTorch and JAX
Best agent passes 75% of migrations while others perform worse, showing migration remains unreliable.