Selective tie learning reduces spurious correlations in preference optimization
Original title: Selective Tie Learning: Mitigating Spurious Correlations in Preference Optimization
Largest predicted reductions are first-order optimal among equal-size tie subsets, and a reward-margin proxy can stand in when influence is costly.
A paper studies how ties, equally preferred responses, can reduce spurious correlations learned during preference optimization, and how to choose which ties to add. For log-linear policies, each tie's influence on spurious reliance is derived, and selecting the largest predicted reductions is first-order optimal among subsets of equal size. When spurious features and objective curvature are unavailable, the magnitude of a tie's implicit reward margin is examined as a proxy. Across log-linear policies, neural models, and large language models, the selection rules matched uniform selection's reductions with fewer ties.
- Influence estimates use a counterfactual proxy for spurious reliance.
- Curvature for language models uses a generalized Gauss-Newton approximation.
- The margin proxy is useful only under conditions the paper characterizes.
- Selection consistently needed fewer ties than uniform selection.
From the original article
1 min read at arxiv.orgPreference optimization can learn spurious correlations, and augmenting the preference data with ties (equally preferred responses) can reduce spurious reliance. However, ties differ in their corrective effect, making their selection for augmentation important.
By Christian Moya · arxiv.org
A short excerpt; the article and its images belong to the publisher. Summary, key facts and assessment above are AI-written from the article and can be wrong.