30
30
30
30
30
30
30
30
30
30
30
Scores Alone Do Not Prove Discovery: The Discovery Certification Protocol for Auditing AI Research Agents
Researchers introduce the Discovery Certification Protocol, an auditable test framework that verifies whether AI research agents' claimed discoveries are genuine.
The Discovery Certification Protocol (DCP) converts AI research agents' discovery claims into executable recovery and feedback tests organized as gated audits. Controlled audits in SQLite optimization and virtual catalyst control produced zero recoveries in 96 episodes, with an upper bound of 0.0468. A deterministic, LLM-free verifier reproduces audit decisions from frozen evidence, giving AI research a common evidence language for outcomes, alternative routes, and feedback effects.
30
30
Why don't machine learning research agents overfit?
Amazon researchers explain why ML research agents avoid benchmark overfitting, attributing generalization to compressibility of successful strategies.
Amazon Science summarizes the paper "What fits (into few tokens) doesn't overfit: Compression and generalization in ML research agents," which investigates why benchmark hill-climbing loops, whether run by human communities or LLM research agents, do not produce rampant overfitting. The explanation formalizes Occam's razor via a counting argument: successful ML strategies are highly compressible, so short descriptions lack room to memorize benchmark data and must capture real structure. LLM-based agents, being resettable and controllable, allow this hypothesis to be tested empirically.
Hacker News · security32
30
30
30
30