Harness-delta method finds much agent-harness gain is overfitting
In 21 significant settings, overfitting and test-time scaling explain 54% and 14% of the gain on average.
Stories per day · 30 days
1 RSS
5 stories, newest first · one card per story
Stories tagged #overfitting
TextReg: Mitigating Prompt Distributional Overfitting via Regularized Text-Space Optimization
TextReg curbs prompt overfitting, gaining up to 16.5% out-of-distribution accuracy over REVOLVE.
Why don't machine learning research agents overfit?
Amazon researchers explain why ML research agents avoid benchmark overfitting, attributing generalization to compressibility of successful strategies.
Data Scarcity and Model Sparsity: Mixtures-of-Experts Overfit More to Repeated Data
Study finds Mixture-of-Experts models overfit faster than dense Transformers under repeated training data, with degradation tied to total parameter sparsity.
Recreating Minecraft Is Not a Benchmark
Opinion piece argues viral demos like one-prompt Minecraft recreations are overfit 'demo-benchmarks' measuring preparation, not true model capability.
2