Direct preference alignment methods, such as DPO, are widely adopted to fine-tune large language models toward human preferences because of their computational efficiency. However, they suffer from a known issue called likelihood displacement: when the margin between preferred and rejected responses is small, optimization can inadvertently degrade overall model quality rather than improve it.
To address this, the authors propose ComPO (Comparison-based Preference Optimization), a zeroth-order method that avoids directly optimizing a differentiable loss over preference pairs. Instead, it extracts directional signals through comparison oracles, an approach related to gradient-free optimization techniques. The authors provide theoretical convergence guarantees for a basic offline scheme, under assumptions of smoothness, gradient sparsity, and compatibility between the oracle and a latent objective.
The paper also introduces an online variant, online ComPO, which keeps the offline comparison mechanism while leveraging unlabeled generations from the current policy to control reverse-KL divergence relative to a reference policy. A performance guarantee is established for a basic constrained scheme, conditional on local coverage and in-distribution pairwise reward accuracy — building on a coverage-based perspective of preference fine-tuning.
Experiments across Mistral, Llama, Gemma-2, Qwen3, and Gemma-3 models show improvements over existing direct alignment baselines, including length-controlled win rates. Pair-level diagnostics offer evidence consistent with mitigated likelihood displacement. The work represents a solid technical contribution within the ongoing effort to refine RLHF-style methods beyond DPO, though it is more incremental research than an immediate shift in industrial practice.