A Zeroth-Order Paradigm for LLM Preference Alignment
Researchers propose ComPO, a zeroth-order comparison-based preference alignment method with convergence guarantees that mitigates likelihood displacement in LLMs.
ComPO extracts directional information from preference pairs via comparison oracles instead of optimizing a differentiable preference loss, addressing likelihood displacement in direct alignment methods. The paper establishes convergence guarantees for the offline scheme and introduces an online variant with reverse-KL control using unlabeled policy generations. Experiments on Mistral, Llama, Gemma-2, Gemma-3, and Qwen3 models show improvements over existing direct alignment methods, including length-controlled win rates.
- ComPO is a zeroth-order alignment method built on comparison oracles
- Offline scheme has convergence guarantees; online variant adds reverse-KL control
- Evaluated on Mistral, Llama, Gemma-2, Qwen3, and Gemma-3 with improved win rates
- Pair-level diagnostics support mitigation of likelihood displacement
From the original article
1 min read at arxiv.orgDirect preference alignment methods are widely used to align large language models (LLMs) with human preferences because of their computational and memory efficiency. However, likelihood displacement motivates alternative ways to extract information from preference pairs with small likelihood margins.
By Peter Chen · arxiv.org
A short excerpt; the article and its images belong to the publisher. Summary, key facts and assessment above are AI-written from the article and can be wrong.