Re-Search tests whether LLMs can rediscover published research ideas
On 200 ICML 2026 papers, six LLMs make nearly all progress in the first 30 of 100 yes-or-no questions.
Stories per day · 30 days
5 RSS
Relatedbenchmarks9benchmark7autoresearch2jailbreak2medical-ai2peer-review2selection-bias2agents1anthropic1attack-success-rate1behavioral-finance1benchmark-maintenance1calibration1classification1claude1clinical-ai1clinical-benchmark1coding-agents1
71 stories, newest first · one card per story
Stories tagged #llm-evaluation
Veritas harness auto-generates agent-task verifiers that beat seven alternatives
With the model fixed, generated verifiers matched official verdicts better than seven harnesses on all four execution-scored benchmarks.
Benchmark plants 100 errors in psychology papers to test AI review
The best system found 71 errors and the pooled systems found 93; seven missed errors were all omissions.
THEATRON benchmark tests persona persistence in long LLM decisions
Four LLMs show persona-driven shifts that often fail to become stable, context-sensitive policies over multi-month financial trajectories.
Compaction beats full transcript in memory
Compaction scores 71.4 percent vs 64.6 percent with 34 percent fewer tokens on LoCoMo.
Siris tests whether LLMs spot invalid simulator recommendations
On LLM serving design problems, spotting a simulator limitation did not guarantee the right design, and a review turn added new errors.
Terminal-Bench 2.1 repairs improve coding agent benchmark accuracy
Benchmark maintenance debt leads to issues in 30% of tasks with accuracy gains over 10 percentage points after repairs.
LLMs still leak real-item knowledge in low-knowledge persona simulations
Across five API endpoints from two model families, explicit ignorance prompts still left a favourable-item real-twin rating gap.
Surge AI benchmarks agents on 50 professional tax assignments
Claude Sonnet 5.5 leads DAYJOB: Tax at a 10.8% pass rate under a 95% rubric threshold.
Rubricon generates LLM rubrics without human preference labels
Rubric scores align 9% more with expert rubrics than the best compared synthetic method, and lift Qwen3.5-9B on HealthBench.
NPC-RubricBench shows LLM cancer-care errors reverse by phase
Across 405 patients, strong diagnosis did not carry into later care, and prompt style traded quality against faithfulness.
Jev-style models still shift answers when options are shuffled
On 1,700 questions, Jev accuracy barely changes, but its cumulative failure set grows 27-29% versus the original order.
Fine-tuned Inception v3 beats GPT-6 Astra and Claude on skin cancer
On 200 biopsy-proven MILK10k lesions, the CNN named the diagnosis for 125 cases versus 84 and 85 for the LLMs.
1
Pydantic schema prompts fail differently across nine LLM setups
Of 3,997 attempts, fenced JSON was the top failure, and only 48.0% of 681 fenced cases passed after fence removal.
Mycelium meta-study maps hallucination detection across benchmarks
The authors sort benchmarks by provenance, typology and granularity and propose Span Coverage F1.
Foyer study: logistic regression beats LLMs on inbox routing
On about 200 labels, logistic regression led every field, while the most accurate system ignored 18% of emails needing action.
LGMT tests whether LLMs contradict equivalent logical queries
Across 3,091 metamorphic groups, six widely used LLMs showed inconsistency rates from 20.84% to 29.44%.
PHYRD benchmark tests LLMs on reproducing and extending physics
Across five models, Reproduce accuracy was 49.21%–68.25% and closed-book Discover was 18.60%–74.42%.
Open Language Models Perform on Spanish Medical Residency Exam
Larger open models rank in the top 600-3000 candidates on the Spanish MIR exam but abstention does not improve outcomes and images are not always used.
Paper measures how AI debates express and organize values
Generated debates often invoke as many values as human references, but spread them more diffusely across turns.
RTL repair scores can miss whether models used the broken design
A new 459-case benchmark finds simulation and formal scoring can disagree, and specification-only baselines swing widely.
DecisionPivot benchmark separates decision accuracy from confidence
Two 27B models only 2.6 points apart on the reference track score 70.1% and 44.2% on contrast groups.
WORTH pilot tests models on mystery evidence discovery
In 37 scorable Episode 02 runs, 31 chose the same wrong participants; adjacent clue types were selected 25 times and 0 times.
Paper proposes ASR_STEP to discount iterative LLM jailbreaks
On JailBreakBench, discounting victim-model calls lowers GCG by 7.0 to 8.7 points and PAIR by 3.4 to 12.