Jev-style models still shift answers when options are shuffled
Original title: Understanding Jev-Style System One Models: Does Structure Reduce Selection Bias?
On 1,700 questions, Jev accuracy barely changes, but its cumulative failure set grows 27-29% versus the original order.
A paper tests whether Jev-style models that score supplied candidates reduce selection bias. Options are shuffled while question and candidate content stay fixed, across two Jev versions, twelve reproductions and six generative LLMs. On 1,700 questions, Jev accuracy varies by less than two percentage points, yet its cumulative failure set expands 27-29% relative to the original order, with first failures continuing into the eighth presentation. Voting gives only modest gains, and even unanimous answers are frequently wrong.
Unconfirmed: Findings come from the authors' shuffle test on two benchmarks, not a broader independent study.
- The loss of initial successes is larger than under fixed-order repetition.
- Jev shows no general advantage over LLM baselines in preserving correct choices.
- Authors say stable accuracy alone cannot show resistance to selection bias.
- AI models
- Jev
From the original article
1 min read at arxiv.orgSelection bias in large language models (LLMs) can persist despite instructions to ignore presentation order or prompts that elicit reasoning. Recent efforts to reproduce Jev-style System One behavior have produced a growing family of models that predict probabilities directly over supplied candidates.
By Mingyang Song · arxiv.org
A short excerpt; the article and its images belong to the publisher. Summary, key facts and assessment above are AI-written from the article and can be wrong.