| title | CroCo: Cross-Lingual Contrastive Preference Tuning on Self-Generations |
|---|---|
| authors | Mike Zhang, Ali Basirat, Desmond Elliott |
| date | 2026-05-25 |
| venue | arXiv (preprint, not peer reviewed) |
| id | arXiv:2605.26293 |
| upvotes | 3 (Hugging Face Daily Papers, 2026-11-01) |
| links | abs pdf hf code |
| abstract | Prior work establishes that controlled contrastiveness between self-generated responses from large language models, set via reward scores, improves downstream preference tuning in English. We extend this method to multiple languages and evaluate two models across a total of 14 high and low-resource languages on a diverse set of tasks. Our central finding is that cross-lingual contrastive preference tuning on self-generations (CroCo) transfers without language-specific preference annotation. A reward model trained on English preferences (atop a multilingual base) produces useful within-language rankings across most languages, and pairing in either a monolingual or multilingual setting improves over each model on the majority of setups while preventing the catastrophic forgetting of supervised fine-tuning. We observe that the gains require on-policy data. Off-policy responses reduce the benefit and online preference optimization fails to improve over the offline variant. Specifically, on structured tasks, our method matches or exceeds the base in 6/7 languages for EuroLLM-9B and 4/7 settings for Aya-3B. On open-ended generation, both tuned models win against their respective base across 11 evaluated languages. Overall, we show promising directions for multilingual preference tuning. |
cite
@misc{zhang2026croco,
title={CroCo: Cross-Lingual Contrastive Preference Tuning on Self-Generations},
author={Mike Zhang and Ali Basirat and Desmond Elliott},
year={2026},
eprint={2605.26293},
archivePrefix={arXiv},
url={https://arxiv.org/abs/2605.26293}
}