A Zeroth-Order Paradigm for LLM Preference Alignment

ComPO is a zeroth-order preference alignment method that uses comparison oracles instead of differentiable preference losses to mitigate likelihood displacement; it offers convergence guarantees for an offline scheme and a reverse-KL-controlled online…

Comparison-based Preference Optimization (ComPO) is a zeroth-order alignment method that extracts directional information from preference pairs with small likelihood margins via comparison oracles, rather than directly optimizing a differentiable preference loss. It targets likelihood displacement, a failure mode that motivates alternatives to direct preference losses. The authors prove convergence guarantees for the offline scheme, assuming smoothness, gradient sparsity, and oracle-objective compatibility, and provide performance guarantees for a constrained online variant that uses unlabeled policy generations for reverse-KL control; a coverage-based analysis supports the constrained scheme. Experiments on Mistral, Llama, Gemma-2, Gemma-3, and Qwen3 show improvements over existing direct alignment methods, including length-controlled win rates, with pair-level diagnostics consistent with mitigating likelihood displacement. The two source reports (Hugging Face daily papers, 2026-09-15; arXiv cs.AI/cs.LG/cs.CL, 2026-09-16) agree on all substantive claims.

• ComPO (Comparison-based Preference Optimization) is a zeroth-order preference alignment method that extracts directional information from preference pairs with small likelihood margins using comparison oracles, without directly optimizing…
• The method is motivated by likelihood displacement, a limitation of direct preference alignment methods.
• The offline scheme has proven convergence guarantees, which assume smoothness, gradient sparsity, and oracle-objective compatibility.
• The online variant is constrained and uses unlabeled policy generations for reverse-KL control; it has performance guarantees supported by a coverage-based analysis.
• Experiments were run on Mistral, Llama, Gemma-2, Gemma-3, and Qwen3 models and show improvements over existing direct alignment methods, including length-controlled win rates.
• Pair-level diagnostics are consistent with mitigation of likelihood displacement.
• Both sources agree on the model set (Mistral, Llama, Gemma-2, Gemma-3, Qwen3) and all substantive claims; no discrepancies were found.