Beyond LLM Judge: Mechanical Verification for Agentic Autoformalization
Mechanical checks plus LLM review cut errors when verifying Lean formalizations of NIST math functions.
The paper proposes verifying Lean autoformalizations with LLM semantic review plus mechanical numerical and vacuity checks, rather than relying only on an LLM judge. Applied through an Extract-Verify-Proof pipeline, the method constructs the Formalized Library of Mathematical Functions, a partial formalization of NIST's Digital Library of Mathematical Functions. On three labeled datasets it achieves the lowest accepted-error rate among evaluated methods, and expert review also surfaces translation and source defects in existing datasets.
- Verification combines LLM semantic review with numerical and vacuity checks.
- The pipeline builds a partial Lean formalization of NIST's DLMF.
- It records the lowest accepted-error rate on three labeled datasets.
- Expert review also finds translation and source defects in existing datasets.
- Code for the Chapter 7 error-function release is public.
- Organizations
- NIST
From the original article
1 min read at arxiv.orgWe introduce a novel method for verifying Lean formalizations that goes beyond LLM-judged faithfulness. Unlike most verification methods, which rely on LLMs or trained models, our method combines LLM-based semantic review with mechanical numerical and vacuity checks, providing checkable feedback and detecting errors missed by LLM judges.
By Anna Mei · arxiv.org
A short excerpt; the article and its images belong to the publisher. Summary, key facts and assessment above are AI-written from the article and can be wrong.