Audit says CAGE 4 defender scores measure less than assumed
Busy-agent handling, unseeded policies, and unpublished scoring choices make published CAGE Challenge 4 comparisons hard to reproduce.
Stories per day · 30 days
2 RSS
Relatedbenchmark2benchmarks2ai-agents1cage1coding-agents1cyber-defense1jailbreaks1language-models1llm-agents1ml-artifacts1ml-research1pretraining-seeds1reinforcement-learning1reprobridge1scientific-discovery1seer1
17 stories, newest first · one card per story
Stories tagged #reproducibility
ReproBridge lifts ML reproduction success from 76.4% to 92.6%
Across 1,029 tasks from 387 experiments, refined artifacts also cut mean API spending by 19.7%.
Pretraining seeds affect jailbreak success more than data recipes in language models
Identical pretraining runs with different seeds produce models with varying susceptibility to 91 jailbreak templates.
LLM agents reproduce 77.4% of claims in SEER cancer papers
A robustness pipeline flags potentially consequential analytical choices in 9.5% of fully reproduced findings for expert review.
Dating the Model: Hidden Dates in System Prompts Affect LLM Evaluation
Hidden dates injected into system prompts shift LLM scores by up to 14% and reorder leaderboards.
Training Witnesses: Trusting the Training without Trusting the Trainer
Witnesses lets trainers certify language-model runs so others can audit training without full reproduction.
Where the Numbers Come From: Auditing Evaluation in Provenance-Based Intrusion Detection
Audit finds provenance IDS scores overstate alarm precision; ThreaTrace labeled only 7 of 994 alarms as attacks.
How Reproducible Are Evaluation Conclusions? A Self-Audit of LLM-Inferred Prompt Structure
A self-audit shows small LLM evaluation tables reliably flag the worst models but not the best.
How UK AISI and EvalEval Are Making Benchmark Results Reproducible
UK AISI and EvalEval are working to make AI benchmark results reproducible.
Gricea: An Open Science Platform for Conversational AI Research
Gricea packages conversational AI studies as runnable, shareable research artifacts, replicating 93% of eligible CUI 2026 paper configurations.
Stanford Researchers Release Paper2Agent: Turning Research Papers Into AI Agents That Reproduce Results and Run on New Data
Stanford researchers released Paper2Agent, a Nature-published pipeline that turns research papers into MCP servers agents can execute.
3
OPEN-1B: A Fully Auditable Training Run
Open-1B releases a 1B-parameter model with bitwise-reproducible training, letting independent auditors verify every step of the run on commodity hardware.
Forging Tree-Ring: Reproducing and Instrumenting Black-Box Semantic Watermark Forgery
Reprompt watermark forgery reproduces on Stable Diffusion XL using free-tier T4 GPUs, with forged images accepted by the genuine detector 5 of 6 times.
IdeaAMBIG: Benchmarking Implementation-Critical Gaps in Research-Idea Specifications
IdeaAMBIG benchmark with 660 instances measures whether LLMs can spot and fix underspecified research-method details for faithful implementation.
IdeaAMBIG: Benchmarking Implementation-Critical Gaps in Research-Idea Specifications
IdeaAMBIG benchmark of 660 specification-gap instances shows LLMs localize implementation-critical research gaps poorly, with best model at 9.6% defect recovery.
Scores Alone Do Not Prove Discovery: The Discovery Certification Protocol for Auditing AI Research Agents
Researchers introduce the Discovery Certification Protocol, an auditable test framework that verifies whether AI research agents' claimed discoveries are genuine.
What We Learned by Reproducing 2,200 papers from ICML
Hugging Face shares lessons from openly reproducing 2,200 ICML 2026 papers, examining reproducibility and open implementation practices in machine learning research.