30
30
30
30
30
30
30
30
30
30
30
30
30
Scores Alone Do Not Prove Discovery: The Discovery Certification Protocol for Auditing AI Research Agents
Researchers introduce the Discovery Certification Protocol, an auditable test framework that verifies whether AI research agents' claimed discoveries are genuine.
The Discovery Certification Protocol (DCP) converts AI research agents' discovery claims into executable recovery and feedback tests organized as gated audits. Controlled audits in SQLite optimization and virtual catalyst control produced zero recoveries in 96 episodes, with an upper bound of 0.0468. A deterministic, LLM-free verifier reproduces audit decisions from frozen evidence, giving AI research a common evidence language for outcomes, alternative routes, and feedback effects.
30
30
30
ImpossibleRubrics: Stress-Testing Generated Rubrics as Reward Signals
ImpossibleRubrics benchmark shows LLM-generated rubric reward signals are exploited 8-26% of the time by adversarial answers on impossible tasks.
ImpossibleRubrics is a benchmark of 169 impossible tasks across six impossibility categories, each paired with a verifiable oracle certificate, plus 48 answerable controls, for stress-testing LLM-generated rubrics used as reward signals. Eleven rubric generators were exploited 8-26% of the time on an unbiased 150-task cut and up to 36% on a stress cut, while a certificate-faithful rubric scored 0%. A single generic 'be decisive, penalize hedging' rubric was exploited 64% of the time, suggesting tailored criteria can reveal which claims attackers should fabricate.
45
30
30
30
30
30
30
30
30
30
30
30
30
30
30
30
Agent Harness vs Agent Framework vs MCP: Which Layer Owns the Loop, State, Tools, Permissions, and Recovery
Architecture explainer separates agent harnesses, frameworks, and MCP by which layer owns the loop, state, permissions, and recovery.
The article distinguishes agent harnesses (OpenAI Codex, Claude Agent SDK), which own the execution loop, sandbox, permission model, and recovery; frameworks (LangGraph, OpenAI Agents SDK, Microsoft Agent Framework), which supply composable primitives; and MCP, a stateless JSON-RPC wire protocol governed by the Linux Foundation's Agentic AI Foundation since December 2025. An ownership matrix maps the execution loop, state, tool transport, permissions, recovery, sandboxing, and multi-agent orchestration to each layer. The 2026-07-28 MCP specification made the protocol fully stateless, retiring the initialize handshake and session headers.
35
Why don't machine learning research agents overfit?
Amazon researchers explain why ML research agents avoid benchmark overfitting, attributing generalization to compressibility of successful strategies.
Amazon Science summarizes the paper "What fits (into few tokens) doesn't overfit: Compression and generalization in ML research agents," which investigates why benchmark hill-climbing loops, whether run by human communities or LLM research agents, do not produce rampant overfitting. The explanation formalizes Occam's razor via a counting argument: successful ML strategies are highly compressible, so short descriptions lack room to memorize benchmark data and must capture real structure. LLM-based agents, being resettable and controllable, allow this hypothesis to be tested empirically.
Hacker News · security32
30
30
30
30
30
30