More Choices, Fewer Decisions: Ordinal-Scale Bias in JEV-like Direct-Decision Models
JEV-like models underuse ordinal label scales, covering only 67–76% of gold support across 36 datasets.
The study finds that JEV 1.13 and three open KEV direct-decision models compress ordinal decision scales. On ANLI, JEV assigns 38.8% of predictions and 51.3% of errors to Neutral despite 74.95% accuracy and nearly balanced labels. Across 36 ordinal datasets, decisions use 67–76% of effective gold support, falling to 26–75% at K=14. BA-LoRA post-training raises utilization from roughly 47% to 86%.
- JEV puts 38.8% of ANLI predictions and 51.3% of errors on Neutral.
- Ordinal decisions use 67–76% of gold support across 36 datasets.
- At K=14, scale utilization falls to 26–75%.
- BA-LoRA raises utilization from about 47% to 86%.
From the original article
1 min read at huggingface.co
Direct-decision models turn text into low-latency structured labels and scores, making them attractive for classification and automatic evaluation. Yet reliability requires more than accuracy: a model must also use the ordinal decision scale supplied by the user faithfully. We analyze JEV~1.
By Tianxiang Gao, Jinzhe Li, Zhiyuan Li · huggingface.co
A short excerpt; the article and its images belong to the publisher. Summary, key facts and assessment above are AI-written from the article and can be wrong.