LGMT tests whether LLMs contradict equivalent logical queries
Original title: LGMT: A Tool for Testing the Logical Reasoning Reliability of LLMs
Across 3,091 metamorphic groups, six widely used LLMs showed inconsistency rates from 20.84% to 29.44%.
LGMT is an open-source tool that looks for self-contradiction in LLM logical entailment. From a premise-conclusion test it generates logically equivalent follow-ups and checks whether predictions agree, without ground-truth labels. It ships 20 built-in metamorphic relations, allows custom relations verified by the Z3 prover, and runs from a CLI or web studio, including an offline mock mode that needs no API key.
Unconfirmed: The paper does not name the six LLMs in the abstract.
- The bundled benchmark contains 3,091 metamorphic groups.
- Reported inconsistency rates span 20.84% to 29.44%.
- Custom relations are verified with the Z3 theorem prover.
- The public repository is named lgmt-tool on GitHub.
From the original article
1 min read at arxiv.orgLarge Language Models (LLMs) increasingly serve as reasoning components in modern software systems, yet a model that answers a logical query correctly can contradict itself when the same problem is restated in an equivalent form. Static benchmarks evaluate queries in isolation and therefore fail to expose such inconsistencies.
By Zenghui Zhou · arxiv.org
A short excerpt; the article and its images belong to the publisher. Summary, key facts and assessment above are AI-written from the article and can be wrong.