PHYRD benchmark tests LLMs on reproducing and extending physics
Original title: PHYRD: A Living Benchmark for Reproducing and Extending Theoretical Physics Research
Across five models, Reproduce accuracy was 49.21%–68.25% and closed-book Discover was 18.60%–74.42%.
PHYRD is a living theoretical-physics benchmark with 63 Reproduce and 43 Discover problems. Reproduce checks derivation of published results; Discover checks consequences of changed theoretical conditions. FormulaForest builds tasks with reference solutions and subquestion rubrics. On five models, whole-problem accuracy was 49.21% to 68.25% on Reproduce and 18.60% to 74.42% on closed-book Discover. Tools helped four models finish more Discover items, but conceptual errors remained. FormulaForest data raised a Qwen-35B-A3B model from 61% to 72% on OpenAI’s FrontierScience.
Unconfirmed: Scores are the authors’ reported results; model identities beyond Qwen-35B-A3B are not named.
- The set has 63 Reproduce problems and 43 Discover problems.
- FormulaForest supplies reference solutions and subquestion-level rubrics.
- Tool use increased Discover completions for four models.
- On FrontierScience, Qwen-35B-A3B rose from 61% to 72%.
- Organizations
- OpenAI
- AI models
- Qwen-35B-A3B
From the original article
1 min read at arxiv.orgLarge language models and agents have shown significant power in advanced physical research. In this paper, We introduce PHYRD, a living benchmark for theoretical-physics reasoning with 63 Reproduce and 43 Discover problems. Reproduce tests derivation of published results; Discover tests consequences of modified theoretical conditions.
By Qituan Zhang · arxiv.org
A short excerpt; the article and its images belong to the publisher. Summary, key facts and assessment above are AI-written from the article and can be wrong.