What Does an LLM-Agent Leaderboard Rank Actually Compare?
A methodological study shows close LLM-agent leaderboard rank gaps on SWE-bench and similar benchmarks often do not support superiority claims.
The paper defines an estimand-aware pairwise procedure for comparing agents, checking common support and applying explicit uncertainty rules and practical margins. Across SWE-bench, AgentRewardBench, and tau2-bench, close rank differences are frequently unresolved, and proxy labels or utility rules can change which system is selected. The authors argue a leaderboard score summarizes a released evaluation but does not by itself justify pairwise superiority conclusions.
- Pairwise agent comparisons require estimand, common support, and uncertainty rules
- Close ranks on SWE-bench, AgentRewardBench, and tau2-bench often unresolved
- Proxy labels and utility rules can flip which system is selected
- DataAgentBench and Open Agent show limits of coarse public records
From the original article
1 min read at arxiv.orgAn LLM-agent leaderboard invites a familiar inference: an agent ranked above another is the better agent. Public evaluation logs may not support that conclusion when systems differ in task mixture, label source, release detail, or cost rule. We study what leaderboard scores estimate and when they justify pairwise superiority conclusions.
By Wei-Jung Huang · arxiv.org
A short excerpt; the article and its images belong to the publisher. Summary, key facts and assessment above are AI-written from the article and can be wrong.