Presentation sensitivity audit shows Jev, Clef, Perplexity Decider vary with framing
Original title: Presentation Sensitivity in Typed Decision Models for Subjective Text Evaluation
Jev, Clef, and Perplexity Decider exhibit position and endorsement biases across humor, helpfulness, and summarization datasets.
The paper audits three typed decision models on six datasets totaling 9,000 items with 702,000 judgments under various presentation interventions. Findings include position advantages, selection changes, and framing effects reducing agreement. Repeatability and robustness are key properties for reliable evaluators.
- Jev shows 6.11 pp second-position advantage on summaries
- Perplexity changes selected humor on 40.40% pairs with endorsement reversal
- Clef summary agreement drops 23.36 points with rejection framing
- Rating formats shift scores but preserve rankings
- Repeatability, robustness, and human alignment are distinct properties
From the original article
1 min read at arxiv.orgTyped decision models offer structured choices, scores, and probabilities for low-cost automated evaluation. We present a controlled audit of presentation sensitivity in Jev, Clef Flash, and Perplexity Decider across native choice and score interfaces.
By Jatin Agrawal · arxiv.org
A short excerpt; the article and its images belong to the publisher. Summary, key facts and assessment above are AI-written from the article and can be wrong.