PORT reweights tokens to align models on noisy preferences
Original title: PORT: Preference Optimization via Robust Token-Level Reweighting
The method uses token log-probability CDFs as a forward-only proxy and beats sequence-level baselines under label noise.
Researchers propose PORT, a preference-optimization method that reweights tokens rather than treating whole responses as uniformly correct or incorrect. It uses the empirical CDF of token log-probabilities as a forward-pass proxy for gradient-norm penalization, aiming to suppress corrupted tokens without an extra backward pass. The authors report a bias bound and say experiments across noise settings improve robustness over sequence-level baselines. Code is at github.com/osu-srml/PORT.
Unconfirmed: Robustness gains are author-reported; the paper does not name the models or noise rates in this abstract.
- Existing robust methods mostly score entire responses as correct or incorrect.
- PORT suppresses tokens using the CDF of token log-probabilities.
- Authors say it reduces gradient bias from noisy preference labels.
- Code is published at github.com/osu-srml/PORT.
From the original article
1 min read at arxiv.orgPreference optimization has become a central approach for aligning large language models with human values, but its effectiveness depends heavily on the quality of preference annotations. In practice, preference data is often noisy due to annotation errors and ambiguity.
By Ding Zhu · arxiv.org
A short excerpt; the article and its images belong to the publisher. Summary, key facts and assessment above are AI-written from the article and can be wrong.