Option order affects answers in non-autoregressive LLM
Original title: Maya a1: Do the Options Read the Model, or Does Their Position? Order Sensitivity in a Non-Autoregressive Decision Model
Order sensitivity causes 25.8% answer flips in Laya model; shuffling during fine-tuning cuts the rate to 7.2%
The paper studies order sensitivity in Laya, a 421M-parameter non-autoregressive model for multiple-choice questions. Rotating options changed answers for 25.8% of 951 questions, with accuracy differing by 10.1 points based on order. Fine-tuning with option shuffling reduced flips from 11.4% to 7.2%. A parallel layout in training cut flips to 0.5%.
- Flip rate of 25.8% on 951 questions
- Accuracy differs by 10.1 points by option order
- Shuffling responsible for 4.2-point improvement
- Parallel layout yields 0.5% flip rate
- AI models
- Laya
From the original article
1 min read at arxiv.orgLaya answers typed multiple-choice questions in one forward pass by scoring a hidden state at one [MASK] marker per option. We re-engineer the checkpoint (421M parameters; a 2-layer option head with no positional tensor; a documented but unused parallel layout) and test whether the answer depends on where an option is listed.
By Mohammed Nayeem Khan · arxiv.org
A short excerpt; the article and its images belong to the publisher. Summary, key facts and assessment above are AI-written from the article and can be wrong.