BestRand Reduces Benchmark Compression Errors
Original title: Random Is Almost Enough? Stable and Low-Cost Benchmark Compression via BestRand
BestRand cuts high-error risk by 50.2% and estimation error by 16.4% at 51.3% of competitor construction cost on 70+ benchmarks.
Random benchmark subsets yield mostly small errors but suffer from a long-tailed distribution of large errors. BestRand preserves random sampling efficiency while mitigating instability. Theory shows most subsets suffice for accurate full-benchmark estimates. Validation across text, multimodal, agent, and VLA benchmarks demonstrates reduced OOD errors.
- Analytical error distribution shows long-tailed risks
- BestRand reduces high-error tail by 50.2%
- Estimation error reduced by 16.4% on average
- Tested on 70+ benchmarks including VLA
- Code and data available on GitHub
- Organizations
- George-Ling3
From the original article
1 min read at huggingface.co
Benchmark compression reduces the often prohibitive time and cost of model evaluation by evaluating models on a small benchmark subset that closely matches full-benchmark results.
By George Ling, Shanshan Zhong, Lijin Yang · huggingface.co
A short excerpt; the article and its images belong to the publisher. Summary, key facts and assessment above are AI-written from the article and can be wrong.