DecisionPivot benchmark separates decision accuracy from confidence
Original title: DecisionPivot: A Diagnostic Benchmark for Decision Models in LLM Systems
Two 27B models only 2.6 points apart on the reference track score 70.1% and 44.2% on contrast groups.
DecisionPivot is a diagnostic benchmark for models that route, guardrail, or verify outputs in LLM systems. It has 48 tasks and 40,033 items across seven decision roles, including a 7,800-item core track with solver-verified contrast groups and a 32,233-item reference track from 35 public sets. Evaluations of 40 decision models and 14 general models show large gaps in contrast-group accuracy, probability ordering, calibration, and selective coverage that aggregate scores hide.
- Perplexity Decider ranks the positive member first in 92.3% of 2,400 pairs but answers both correctly in 74.1%.
- GLM-5.3 reaches 93.5% contrast-group accuracy at a higher reasoning setting versus 81.5% at lowest effort.
- The core track uses 13 generated task families with independently implemented solvers.
- The 1,296-item GLM-5.3 comparison is reported as a separate subset.
From the original article
1 min read at huggingface.co
Decision models support routing, guardrails, and output verification in LLM systems. However, item accuracy alone cannot establish whether models make correct decisions across controlled variations or assign reliable confidence to their predictions. We introduce DecisionPivot, a diagnostic benchmark comprising 48 tasks and 40,033 items across seven decision roles.
By Xialie Zhuang, Mingcong Lei, Zunhai Su · huggingface.co
A short excerpt; the article and its images belong to the publisher. Summary, key facts and assessment above are AI-written from the article and can be wrong.