One Bias Score, Two Factors: How Mitigations Affect Option Bias in Multiple-Choice Evaluation
A bias score splits into flip rate and positional skew, showing how two mitigations affect multiple-choice LLM evaluation.
The paper decomposes multiple-choice option bias using the pooled selection-rate standard deviation, which equals the product of flip rate and positional skew when options are balanced across positions. Across 41 checkpoints and 12 benchmarks, flip rate tracks accuracy and output uncertainty, while positional skew tracks the option-symbol prior. Prior correction mainly reduces positional skew, and aggregating each ordering with its reversal mainly reduces flip rate. Both narrow accuracy ranges across correct-answer letters, but aggregation can settle questions on the same wrong option, so the authors recommend reporting both factors with accuracy.
- Pooled selection-rate SD factors into flip rate and positional skew.
- Evaluated across 41 checkpoints and 12 benchmarks.
- Prior correction mainly reduces positional skew.
- Order-reversal aggregation mainly reduces flip rate, sometimes onto wrong answers.
- Authors recommend reporting both factors alongside accuracy.
From the original article
1 min read at huggingface.co
Reordering multiple-choice options can change a model's answer, an effect studied as selection bias (a preference for certain option letters) and as order sensitivity (changes in answers across option orderings). We connect these perspectives through the pooled selection-rate standard deviation (pooled SD), which measures how unevenly option letters are selected across questions and orderings.
By Hyunjong Ok, Minjae Park, Jaeho Lee · huggingface.co
A short excerpt; the article and its images belong to the publisher. Summary, key facts and assessment above are AI-written from the article and can be wrong.