Competence-based eval finds GPT and Qwen fail ordering tasks
Original title: Competence-Based Evaluation of Logical Reasoning in Large Language Models
Even strong models fail to apply the same ordering competence across presentations, despite good accuracy in bounded ranges.
The paper argues that single accuracy scores miss whether models apply the same reasoning competence across a family of problems. Competence-based evaluation uses a parameterized generator and matched answer-preserving presentations. On elementary ordering tasks, six GPT and Qwen models do reasonably well over bounded complexity but fail the competence condition across the tested family. Chain-of-thought and GRPO extend strong performance without resolving all failures.
- A generator scales complexity while keeping the required competence fixed.
- Matched presentations test correctness beyond one formulation.
- Chain-of-thought and GRPO leave some tested failures unresolved.
- Supervised LoRA shows little held-out transfer.
From the original article
1 min read at huggingface.co
Recent advances in large language models (LLMs) have brought impressive performance across diverse tasks; however, shortcomings remain in logical reasoning, including inconsistencies under linguistic variation. To improve the reasoning capacity of LLMs, we must be able to accurately measure their reasoning capacity. Current benchmarks primarily report single accuracy numbers on selected reasoning tasks.
By Jize Jiang, Chengxiang Zhai · huggingface.co
A short excerpt; the article and its images belong to the publisher. Summary, key facts and assessment above are AI-written from the article and can be wrong.