Ramanujan Challenge paper reports an AI pipeline for proof judging
Original title: Ramanujan Challenge Results: AI-Assisted Evaluation Pipeline for Multi-Format Proofs
The pipeline cut false-positive verdicts on natural-language proofs from 23% to 6% versus one-shot LLM judgment.
The authors analyze submissions to the Ramanujan Challenge, a benchmark of research-level number-theory problems that includes unpublished results and open conjectures, and compare AI models, computer algebra systems and formal tools. They report correct submissions conceptually distinct from the official solutions and a proof of one published open conjecture. They also introduce an AI-assisted pipeline that judges natural-language proofs, CAS code and Lean formalizations for semantic and logical correctness. Versus one-shot LLM judgment, false-positive verdicts on natural-language proofs fell from 23% to 6%, and accuracy on supporting code rose from 46% to 58%.
Unconfirmed: The claimed proof of an open conjecture is reported by the authors and not independently confirmed here.
- The benchmark includes unpublished results and open conjectures.
- Formulas for constants can be checked numerically.
- Judgment accuracy on supporting code rose from 46% to 58%.
- The pipeline covers natural language, CAS code and Lean.
- One submitted proof addresses a published open conjecture.
From the original article
1 min read at arxiv.orgAs AI's mathematical capabilities rapidly improve, meaningful benchmarks are shifting from university-level problems and math olympiads to research-level mathematics. The Ramanujan Challenge is a benchmark comprised of research-level problems in number theory, including unpublished results and open conjectures.
By Alan Sorani · arxiv.org
A short excerpt; the article and its images belong to the publisher. Summary, key facts and assessment above are AI-written from the article and can be wrong.