| title | Learning from Noisy Preferences: A Semi-Supervised Learning Approach to Direct Preference Optimization |
|---|---|
| authors | Xinxin Liu, Ming Li, Zonglin Lyu, Yuzhang Shang, Chen Chen |
| date | 2026-04-27 |
| venue | arXiv (preprint, not peer reviewed) |
| id | arXiv:2604.24952 |
| upvotes | 5 (Hugging Face Daily Papers, 2026-11-01) |
| links | abs pdf hf |
| abstract | Human visual preferences are inherently multi-dimensional, encompassing aesthetics, detail fidelity, and semantic alignment. However, existing datasets provide only single, holistic annotations, resulting in severe label noise: images that excel in some dimensions but are deficient in others are simply marked as winner or loser. We theoretically demonstrate that compressing multi-dimensional preferences into binary labels generates conflicting gradient signals that misguide Diffusion Direct Preference Optimization (DPO). To address this, we propose Semi-DPO, a semi-supervised approach that treats consistent pairs as clean labeled data and conflicting ones as noisy unlabeled data. Our method starts by training on a consensus-filtered clean subset, then uses this model as an implicit classifier to generate pseudo-labels for the noisy set for iterative refinement. Experimental results demonstrate that Semi-DPO achieves state-of-the-art performance and significantly improves alignment with complex human preferences, without requiring additional human annotation or explicit reward models during training. We will release our code and models at: https://github.com/L-CodingSpace/semi-dpo |
cite
@misc{liu2026learning,
title={Learning from Noisy Preferences: A Semi-Supervised Learning Approach to Direct Preference Optimization},
author={Xinxin Liu and Ming Li and Zonglin Lyu and Yuzhang Shang and Chen Chen},
year={2026},
eprint={2604.24952},
archivePrefix={arXiv},
url={https://arxiv.org/abs/2604.24952}
}