Search: “overfitting”

Why don't machine learning research agents overfit?

Amazon researchers explain why ML research agents avoid benchmark overfitting, attributing generalization to compressibility of successful strategies.

Amazon Science summarizes the paper "What fits (into few tokens) doesn't overfit: Compression and generalization in ML research agents," which investigates why benchmark hill-climbing loops, whether run by human communities or LLM research agents, do not produce rampant overfitting. The explanation formalizes Occam's razor via a counting argument: successful ML strategies are highly compressible, so short descriptions lack room to memorize benchmark data and must capture real structure. LLM-based agents, being resettable and controllable, allow this hypothesis to be tested empirically.

Hacker News · security·Sep 14, 16:32 UTC · 2d agoAI researchHN 21↑ · 3 comments

Data Scarcity and Model Sparsity: Mixtures-of-Experts Overfit More to Repeated Data

Study finds Mixture-of-Experts models overfit faster than dense Transformers under repeated training data, with degradation tied to total parameter sparsity.

Across models from 80M to 1B active parameters (8.5B total), MoE architectures degrade more rapidly than dense models when training data is repeated, with the effect increasing with sparsity as dictated by total parameters. Dense 80M models tolerate 8x repetition with minimal loss while MoEs suffer at 4x and underperform dense models beyond 32x. Masking-based regularization such as dropout mitigates overfitting, letting MoEs beat dense models even at over 64x repetition, though no method matches all-unique training data. Routing stabilizes early and expert specialization correlates with overfitting to repeated data.

arXiv cs.AI / cs.LG / cs.CL·Sep 10, 17:57 UTC · 6d agoAI research

Recreating Minecraft Is Not a Benchmark

Opinion piece argues viral demos like one-prompt Minecraft recreations are overfit 'demo-benchmarks' measuring preparation, not true model capability.

The author argues that fixed, famous demo tasks (Minecraft builds, SVG pelicans) are trivially optimizable by labs each release cycle, so they no longer differentiate model capability. The piece cites Thinking Machines' Inkling Small scoring within a point of its flagship on the Artificial Analysis Intelligence Index with less than a third of the parameters, and beating it on Humanity's Last Exam, GPQA Diamond, and SciCode. The proposed alternative is rotating or holdout evals such as LiveBench, ARC-AGI's private set, and held-back portions of Humanity's Last Exam.

Hacker News · AI·Sep 6, 14:52 UTC · 10d agoAI researchHN 73↑ · 71 comments

ConvMem: Convolutional Memory for Long-Context Reasoning

Researchers propose ConvMem, a training-free framework treating LLMs as convolutional kernels for parallelizable long-context reasoning beyond fixed context windows.

ConvMem reformulates long-context reasoning as a hierarchical convolution in which the LLM summarizes text segments hierarchically, shortening the reasoning path from a linear chain to a logarithmic tree. It uses configurable strides, skip connections, and multi-kernel convolution to capture evidence, decompose queries, and enable massive parallelization across segments and reasoning threads. On RULER-HotpotQA and RULER-2WikiMultiHopQA it outperforms training-free baselines and avoids the out-of-distribution overfitting seen in RL-trained approaches like MemAgent.

arXiv cs.AI / cs.LG / cs.CL·Sep 9, 16:53 UTC · 7d agoAI research

Foundation Models for Generalizable Semantic and Goal-Oriented Communication

FMSGOC uses vision-language foundation model priors plus diffusion reconstruction to enable generalizable semantic communication at 0.039 bits per pixel for 6G.

FMSGOC targets generalization failures in semantic and goal-oriented communication for 6G by leveraging broad visual-linguistic foundation model priors. A vision-language model selects sparse, goal-aligned semantic anchors while a fine-tuned diffusion model performs masked completion to reconstruct images at the receiver, decoupling what to send from how to reconstruct. On CIFAR-10 it reaches 0.039 bits per pixel with cosine similarity 0.87-0.90 and 0.83-0.86 on unseen ImageNet inputs, outperforming end-to-end baselines at lower bit rates.

arXiv cs.AI / cs.LG / cs.CL·Sep 7, 18:12 UTC · 9d agoAI research

Measuring benchmark optimization in speech recognition

Hugging Face examines how much speech recognition systems overfit benchmarks and how to measure benchmark optimization in ASR.

A Hugging Face post on measuring benchmark optimization in automatic speech recognition, analyzing how model improvements on benchmarks reflect genuine capability gains versus overfitting. It is evaluation methodology research with no direct security impact.

Hugging Face Blog·Aug 21, 00:00 UTC · 27d agoAI research

T1: Terminal Agent Reinforcement Learning for Long-Horizon Tasks

T1, a 122B MoE terminal agent trained with reinforcement learning, reaches 64.0% on Terminal-Bench 2.1, surpassing GPT-5.4 and GLM-5.1 on long-horizon tasks.

T1 is a 122B mixture-of-experts model trained with reinforcement learning to operate a real shell in a cloud sandbox for up to 300+ tool-call turns per task, rewarded by executing each task's own verifier. The recipe combines aggressive warm starts, dense process rewards, TITO construction, and rollout routing replay, cutting the training-to-inference log-probability difference from 0.021 to 0.013 with zero token drift. Training used an out-of-distribution corpus disjoint from Terminal-Bench 2.1. Post-training raised the base model from 43.8% to 64.0% resolved on Terminal-Bench 2.1 and 27.9% on Long-Horizon Terminal Bench.

Hugging Face daily papers·Sep 9, 20:00 UTC · 7d agoAI research1

Measuring Language Transfer in Robot Policies: Adding Greek to a Cosmos3 Vision-Language-Action Policy

Researchers added Greek to the Cosmos3 vision-language-action policy using only machine-rephrased instructions, finding bilingual training reaches roughly two fifths of English performance.

The paper studies localizing the open Cosmos3 vision-language-action robot policy to Greek without architectural changes, using machine-rephrased instructions only. Bilingual training yields a consistent 6.7-7.1 point margin over controls on a 90-task, three-seed evaluation suite, while Greek-only training gains at most 2.7 points. Several common evaluation instruments, including color-histogram metrics and single-goal benchmarks, produced false conclusions, and results were dominated by seed variation. The authors recommend building guaranteed-null baselines and replicating low-resource-language results across seeds.

Hugging Face daily papers·Sep 6, 20:00 UTC · 10d agoAI research

Harnessing CLIP and DINO: An Uncertainty-Aware Cascaded Fusion Network for Generalizable Deepfake Image Detection

UCF-Net fuses CLIP and DINO features with entropy-based uncertainty weighting to improve generalizable deepfake image detection across generators.

Researchers propose UCF-Net, an uncertainty-aware cascaded fusion network that combines CLIP's language-aligned semantic priors with DINO's self-supervised visual-structure priors for deepfake detection. It aggregates hierarchical features across transformer depths via layer-wise expert modules and performs weighted fusion driven by entropy-derived uncertainty. The authors consolidate public deepfake datasets into a unified benchmark of roughly 4 million images plus a cross-generator set of over 8,000 faces from eight recent generators, where UCF-Net achieves the best mean AUC among evaluated methods, though zero-shot transfer remains challenging.

Hugging Face daily papers·Sep 6, 20:00 UTC · 10d agoAI research

When Models Edit Too Much: On the Fidelity of Minimal Code Edits

A 400-task BigCodeBench evaluation shows frontier LLMs widely over-edit code; a preservation instruction cuts excess edits and raises Pass@1 by 2.3 points.

Researchers built an evaluation framework from 400 BigCodeBench problems with injected AST-level corruptions, each with a known minimal patch, to measure over-editing in LLM code repair. Even strong models like GPT-5.5 produce unnecessarily large edits despite high Pass@1. Adding a preservation instruction reduced average excess Levenshtein distance from 0.195 to 0.131, cut added cognitive complexity by 26.6%, and raised Pass@1 by 2.3 points. Reinforcement learning post-training gave the best out-of-domain edit-fidelity trade-off, while supervised fine-tuning overfit to seen corruption patterns.

Hugging Face daily papers·Sep 2, 20:00 UTC · 14d agoAI research1

Your photo could be all AI needs to clone your voice

Help Net Security·Oct 30, 00:00 UTC · Oct 30, 2025AI research