PORT reweights tokens to align models on noisy preferences
The method uses token log-probability CDFs as a forward-only proxy and beats sequence-level baselines under label noise.
Stories per day · 30 days
2 RSS
Relatedalignment1dpo1grpo1lambdamix1language-models1llm-alignment1llm-training1low-rank-adapters1multi-objective1noisy-labels1spurious-correlations1ties1token-reweighting1training-efficiency1
7 stories, newest first · one card per story
Stories tagged #preference-optimization
LambdaMix learns steerable policies with jointly trained low-rank adapters
One training run mixes per-objective adapters so preferences can change at inference without more optimization.
Selective tie learning reduces spurious correlations in preference optimization
Largest predicted reductions are first-order optimal among equal-size tie subsets, and a reward-margin proxy can stand in when influence is costly.
Stochastic DPO graph pruning speeds training about 1.4x
Branch pruning cut training time about 30% versus stock DPO, with slightly lower mean accuracy.
Preference-Guided Adaptation for Open-Vocabulary Semantic Segmentation via Prompt Disagreement
A preference-guided method adapts open-vocabulary segmentation using prompt disagreement, improving MESS results without pixel masks.
Rethinking Critic Learning in PPO: Understanding and Mitigating Value Flattening
Researchers identify Value Flattening in PPO critics for LLM RL and propose SP^3O sparse value supervision, improving Qwen3-Base training.
PLC-DPO: Posterior Label Correction in Noisy and Ambiguous Preference Optimization
PLC-DPO corrects noisy preference labels during DPO by routing each pair as clean, flip, or tie using calibrated policy-reference margins.