A Zeroth-Order Paradigm for LLM Preference Alignment
ComPO is a zeroth-order preference alignment method using comparison oracles to mitigate likelihood displacement across Mistral, Llama, Gemma, and Qwen3 models.
The paper proposes Comparison-based Preference Optimization (ComPO), a zeroth-order alignment method that extracts directional information from preference pairs with small likelihood margins without directly optimizing a differentiable preference loss. The authors prove convergence guarantees for the offline scheme and performance guarantees for a constrained online variant with reverse-KL control. Experiments on Mistral, Llama, Gemma-2, Qwen3, and Gemma-3 show improvements over existing direct alignment methods, including length-controlled win rates, with pair-level diagnostics consistent with mitigating likelihood displacement.
- Likelihood displacement motivates alternatives to direct preference losses.
- Online ComPO uses unlabeled policy generations for reverse-KL control.
- Guarantees assume smoothness, gradient sparsity, and oracle-objective compatibility.
- Coverage-based analysis supports the constrained scheme.
- Products
- ComPO
From the original article
1 min read at huggingface.co
Direct preference alignment methods are widely used to align large language models (LLMs) with human preferences because of their computational and memory efficiency. However, likelihood displacement motivates alternative ways to extract information from preference pairs with small likelihood margins.
By Peter Chen, Xi Chen, Wotao Yin · huggingface.co
A short excerpt; the article and its images belong to the publisher. Summary, key facts and assessment above are AI-written from the article and can be wrong.