Lean compilation overstates faithfulness of statement autoformalization
Original title: Beyond Compilation: Evaluating Faithful Natural-Language-to-Lean Statement Formalization
A tool-augmented agent compiled 87.2% of 227 statements, but only 57.7% passed a two-judge semantic check.
Lean can confirm a declaration is well typed but not that it matches the user’s intent. The authors require compilation plus agreement from two LLM judges of different families. Across eight systems on 227 university statements, the compile-versus-accepted gap ranged from 1.3 to 29.5 points. Human review found most of the agent’s gap was real translation error, while much of the gap for two newer models was judge error.
Unconfirmed: For GPT-6 Astra and Claude Opus 5.5, reviewers classed only 4 of 54 rejections as translation errors.
- The criterion matched the human majority on 91.5% of an independent audit.
- Human review tied 23.8 of the agent’s 29.5-point gap to translation errors.
- GPT-5.2 accepted 83% of later models’ statements versus 21% for one-shot GPT-5.2.
- Lean feedback drove most compilation gain; nearly half failed the semantic test.
- Products
- Lean
From the original article
1 min read at huggingface.co
Lean verifies that a generated declaration is well typed, but cannot check whether it states what the user intended. We measure how far compilation overstates faithfulness in statement autoformalization without canonical Lean targets. Our criterion requires Lean compilation and agreement of two LLM judges from different model families.
By Ke Zhang, Patricio Gallardo Candela, Sudhir Murthy · huggingface.co
A short excerpt; the article and its images belong to the publisher. Summary, key facts and assessment above are AI-written from the article and can be wrong.