Import AI 478 covers math formalization, ProgramBench, and agent escapes
Claude Opus 5.5 scores 18.5% resolved on ProgramBench, up from 0% for earlier models in May.
2
Stories per day · 30 days
2 RSS
Relatedlean5formal-verification3agent-security1dlmf1equivalence-checking1faithfulness1llm-judge1llm-judges1mathematical-functions1mathematics1mathlib1programbench1proof-assistants1research-papers1side-channel1
7 stories, newest first · one card per story
Stories tagged #autoformalization
SemantEq checks faithful Lean 4 statement equivalence
A model-free search hits 100% precision and 65.6% recall; proof auditing accepts none of 104 non-equivalent pairs.
PaperFormalizer formalizes research papers in Lean with the author in the loop
It runs in Claude Code or Codex, pauses on ambiguous claims, and emits an annotated paper plus linked Lean code.
Lean compilation overstates faithfulness of statement autoformalization
A tool-augmented agent compiled 87.2% of 227 statements, but only 57.7% passed a two-judge semantic check.
Beyond LLM Judge: Mechanical Verification for Agentic Autoformalization
Mechanical checks plus LLM review cut errors when verifying Lean formalizations of NIST math functions.
OpenAI’s math solutions aren’t meeting the field’s standards yet
Mathematicians say OpenAI's hundreds of claimed hard-problem proofs miss standards for human understanding.
4
Beyond Solver Verdicts: Generative Reward Models for Autoformalization
Researchers introduce Generative Verification (GenV), a generative reward model achieving 0.961 AUROC in detecting unfaithful autoformalization that preserves solver verdicts.