No AI summary available for this article.
Why It Matters
Direct preference alignment methods are widely used to align large language models (LLMs) with human preferences because of their computational and memory efficiency.
Provenance
Discovered via ArXiv and published by ArXiv.
Key Claims
Original description
Direct preference alignment methods are widely used to align large language models (LLMs) with human preferences because of their computational and memory efficiency. However, likelihood displacement motivates alternative ways to extract information from preference pairs with small likelihood margins. In this paper, we propose and analyze Comparison-based Preference Optimization (ComPO), a zeroth-order alignment method based on comparison oracles. ComPO extracts directional information from these pairs without directly optimizing a differentiable preference loss on them. We establish a converg...
Discovered via ArXiv
Research papers and preprints from arXiv.
Publisher: arxiv.org
ID: http://arxiv.org/abs/2609.19144v1 · Indexed about 1 hour ago