PLC-DPO: Posterior Label Correction in Noisy and Ambiguous Preference Optimization
PLC-DPO corrects noisy preference labels during DPO by routing each pair as clean, flip, or tie using calibrated policy-reference margins.
PLC-DPO addresses reversed, weak, or ambiguous preference labels in Direct Preference Optimization by using the calibrated policy-reference margin as online evidence to route each pair's training signal as clean, flip, or tie. This actively corrects supervision direction and strength rather than only filtering suspicious examples. Across 57 dataset-model-benchmark cells, PLC-DPO achieves the best mean win rate of 60.5 versus 55.5 for the next-best method. Injected-noise and tie stress tests plus human disagreement analysis show the routing stays stable and distinguishes flipped from weakly directional pairs.
- Routes each preference pair as clean, flip, or tie during training
- Calibrated policy-reference margin serves as online routing evidence
- Best mean win rate 60.5 vs 55.5 next-best across 57 evaluation cells
- Stable routing confirmed by injected-noise and human disagreement analyses
From the original article
1 min read at huggingface.co
Direct Preference Optimization (DPO) simplifies alignment through pairwise comparisons but assumes all observed preferences are reliable. Real data often violates this assumption, leading to reversed, weak, or ambiguous labels that cause harmful policy updates.
By Boryeong Cho, Sumyeong Ahn, Se-Young Yun · huggingface.co
A short excerpt; the article and its images belong to the publisher. Summary, key facts and assessment above are AI-written from the article and can be wrong.