PrimaCons benchmark shows Jev decision inconsistencies
Original title: Cross-Primitive Consistency in Jev
PrimaCons reveals cross-primitive consistency violations in Jev, mitigable by CPRC reducing inconsistency by 58.57%.
Jev enables efficient decision-making with LLMs through typed querying primitives. PrimaCons is a four-axis benchmark assessing cross-primitive consistency. Evaluations across three Jev sizes reveal consistency violations. These violations follow systematic patterns predicting decision failure with ROC-AUC of 0.91. The Cross-Primitive Residual Calibrator reduces inconsistency by 58.57% and improves accuracy by 2.97%.
- Consistency violations across all four axes in three Jev sizes
- Cross-primitive discrepancies predict decision failure with ROC-AUC 0.91
- CPRC reduces inconsistency by 58.57% and improves accuracy by 2.97%
- Works under leave-one-domain-out protocol
From the original article
1 min read at huggingface.co
Jev enables efficient decision-making with large language models (LLMs) through typed querying primitives, allowing the same semantic state to be expressed in different formats, such as binary and multiple-choice queries. However, the consistency of decisions across primitives has not been systematically evaluated.
By Xu Liu, Guikun Chen, Kanzhi Wu · huggingface.co
A short excerpt; the article and its images belong to the publisher. Summary, key facts and assessment above are AI-written from the article and can be wrong.