HSPR recovers software-agent benchmark scores from small task subsets
Original title: HSPR: Recovering Software-Agent Benchmark Scores via Hierarchical Posterior Inference
At a 10% task budget on four SWE-bench family banks, it had the lowest MAE on three and the lowest MaxErr on all four.
Hierarchical Score Posterior Recovery (HSPR) estimates a software agent's full benchmark score from a small task subset. It combines source-fitted response factors with uncertain group success rates, then updates posteriors as target outcomes arrive and picks later tasks to cut full-score variance. The final estimate is the posterior median with source-selected calibration. On four SWE-bench family banks, HSPR beat compared methods on error, and macro MAE fell as budgets rose from 5% to 20%.
- A 10% task budget was used on four SWE-bench family task banks.
- HSPR had the lowest MAE on three banks and the lowest MaxErr on all four.
- Dropping group uncertainty or adaptive queries raised macro MAE in controls.
- Macro MAE decreased across budgets from 5% to 20%.
From the original article
1 min read at huggingface.co
Evaluating software agents on a full benchmark is expensive. A small task subset can preserve agent rankings while yielding inaccurate scores. Historical responses provide useful structure, but tasks with identical source responses can have different outcomes for a new target agent.
By Wanyi Chen, Wanqin Shen, Yuanchen Shi · huggingface.co
A short excerpt; the article and its images belong to the publisher. Summary, key facts and assessment above are AI-written from the article and can be wrong.