PhyRubric scores physical commonsense errors in text-to-video models
A judge trained on 3,788 videos tracks human ratings more closely than other open judges and matches Gemini 4 Argon.
Stories per day · 30 days
20 RSS
Relatedbenchmarks21benchmark9llm-agents9multimodal6language-models5world-models5agents4reinforcement-learning4coding-agents3swe-bench3agent-memory2ai-agents2alignment2audio-video2continual-learning2dataset2faithfulness2forward-deployed-engineering2
269 stories, newest first · one card per story
Stories tagged #evaluation
Interviews with 15 builders map forecasting-bot construction problems
Successful competition participants struggled to ground bots in facts, keep reasoning coherent, and judge real performance.
Web-agent harnesses stay vulnerable across prompt and page attacks
WebTrap Farm separates defensive failures from task limits and finds harness design, not only the model, shapes outcomes.
Embodied First Exam tests coding agents on 1,156 robot tasks
GPT-6 Astra in Codex, strongest of eight agents, succeeds on 34% of 456 core tasks.
WorldFoundry catalogs 270 models for world-model evaluation
The open-source stack also lists 74 benchmarks and 62 metrics, separating execution evidence from scores.
ToyBlocks is a controlled world for robot learning
Hugging Face published an Allen Institute for AI post; the available item gives only the project title.
JailBound FOL-Guided Framework Evaluates LLM Jailbreaks
JailBound combines taxonomy-driven prompt generation with FOL-guided dual-branch search, boosting average ASR by 5.1 points across 46 LLMs.
FINCOSMOS evaluates finance agents without fixed tasks, data, or timelines
In 150 sessions no run passed; agents surfaced only 54-64% of untold details, and experts blamed them for 88.33% of failures.
Disagreement among faithfulness evaluation criteria
Different operationalizations of explanation faithfulness disagree on evaluations, with choices justified by explanatory and normative factors.
LLMs falsely reassure users on active NWS severe-weather alerts
LLMs provide false reassurance in 37.7% of replayed NWS alerts without alert access and 44.9% when instructed to disclose blind spot.
PKPS enables benchmark prediction with sparse responses
PKPS represents models to predict benchmarks from sparse unpaired data.
EMBER2024 models cannot estimate executable first-seen dates
Machine learning on EMBER2024 features or raw bytes cannot replace external first-seen metadata for malware evaluation splits.
One 1B decoder gates and writes memory-grounded conversation hints
End-to-end exact match is 0.2619 versus an oracle-gate ceiling of 0.2971 across 28,058 decision points.
Google releases MTG-AD benchmark for inline audio descriptions
The 36-hour set has more than 8,000 descriptions, and end-to-end scoring checks whether narration fits existing video silences.
Prompting versus token probabilities tested on semantic focus
Token probabilities beat chance across sizes, while prompting sometimes wins widely in larger models.
HALO evaluates proactive agents with human annotations
HALO-8K dataset tests 10 models across 400 scenarios, revealing over-proactivity in 61.20% of rejections.
Tutorial covers generative models for music audio
It compares codecs, token models, diffusion, and flow matching, and discusses metrics, listening tests, and legal disputes.
Same-class controls isolate local audio evidence in model answers
Different-class edits change answers 49.7 to 60.9 points more often than matched same-class replacements.
Studied latent world models fail a faithfulness test
Latent rollout consistency checks whether action-conditioned latent prediction matches the environment transition map.
RubricAtlas mines type-level rubrics to score deep research agents
Built from 3,079 Chinese financial reports across 99 types, judge pipelines agreed at kappa 0.71 and ranking tau 0.87.
Framework argues AI agent restart monitors should be judged by decision value, not accuracy
A 90% accuracy monitor can swing from a 5-point loss to a 35-point gain in agent success depending on residual restart budget, an exact analytical result.
Auto-Research Arena benchmarks autonomous language-model research
A sequential search baseline beat several complex harnesses; reused ideas lowered loss under matched FLOPs.
DECE arena compares LLM agents across ten Eurogames
Twelve model-harness setups played 480 three-player matches; 1,032 matches were completed in total.
Co-Design Bench scores coding agents on real user revisions
The strongest of ten endpoints passes 69.9% of criteria but fully satisfies only 26.4% of tasks.