Trace-based judging leads analytics-agent evaluation comparison
On 72 labeled cases the trace-based harness reached F1 0.90, while red-teaming found 24 failures missed by functional tests.
Stories per day · 30 days
2 RSS
Relatedbenchmarks3llm-agents3foundation-models2work-agents2agent-harness1ai-agents1computer-use-agents1computer-using-agents1digital-work1enterprise-analytics1gamecraft-bench1game-generation1inference-cost1llm-judging1long-horizon1multi-domain1persona1post-training1
17 stories, newest first · one card per story
Stories tagged #agent-evaluation
VeriCUA verifies computer-using agents by gathering GUI and code evidence
The framework builds a task rubric, then runs GUI or code checks and rejects unsupported completion claims.
Paper frames work agents as model-plus-harness systems for verifiable tasks
It defines work persistence and openness, five work forms, and model, harness, and joint optimization.
SearchHorizonBench tests whether AI agents can value another search
On 1,632 held-out questions, historical calibration cut Brier error 16.2%, but point estimates explain under 2% of search-gain variance.
Paper frames work agents as systems that deliver verifiable outcomes
It defines work persistence and openness, five work forms, and five capability families for reliable execution.
ParetoRSI-Gym evaluates multi-domain recursive self-improvement
Claude Code with Opus-5 raised equal-weighted generalization scores 9.2% relative, or 2.3 absolute points, over the starting models.
VibeGame builds playable 2D games from prompts with agent review loops
Eight specialized agents separate coding from review, then distill accepted games into reusable skeletons without extra training.
Taste-Bench finds top models weak at long-term agent choices
GPT-5.6 Sol scored 59.7 percent on 502 binary forks; trained Qwen3.6-27B advice more than doubled SWE-bench Pro success.
12
PersonaCUABench: computer-use agents lose 36% completion on persona requests
Nine backbones ran 3,330 times; top satisfaction after full persona completion was 62.7% versus 84.1% on original requests.
ASAP evaluates agent behavior in base models before post-training
The framework keeps the completion interface and scores software-engineering and deep-research tasks across checkpoints.
LiveMACEBench scores LLM agents by process in live markets
Over 30 days, five frontier models' returns often diverged from mechanism-specific capability measurements.
Judged Useless, Queried Anyway: Tool-Using Agents Rarely Turn Their Own Evidence Judgments into Stopping Decisions
Seven tool-using agents almost always call failed retrieval useless but rarely stop unless the harness forces it.
545 Hackers Tested It First. Now XRanges for AI Scores Your Security Agent
CTF.ae's XRanges for AI platform scores autonomous security agents on realistic instrumented targets using coverage, boundaries, exploited, and integrity signals.
An Empirical Study of Harness Design for Coding Agents
Empirical study of coding agent harnesses across 176 settings shows context management prevents overflow failures and bash-only interfaces cut costs for capable models.
An Empirical Study of Harness Design for Coding Agents
Empirical study of 176 matched coding-agent harness configurations shows context management cuts overflow failures while planning and action-space choices trade accuracy against cost.
COBRA-Skills: Contextual Bandit-Guided Evolution for Agent Skill Optimization
COBRA-Skills uses contextual bandits to guide LLM agent skill evolution, cutting optimization cost 55-58% versus SkillOpt while topping six agent benchmarks.
JarvisGUI: Towards Cross-Device GUI Agents with Dynamic Task Composition
JarvisGUI benchmark tests GUI agents on cross-device workflows across Android, Windows, and Ubuntu, revealing major gaps in state transfer and long-horizon reasoning.