| title | Token-Level Off-Policy Learning for Faithful Generation Under Distribution Shift |
|---|---|
| authors | Zitong Huang, Gustavo Lucas Carvalho, Deqing Fu, Robin Jia |
| date | 2026-07-20 |
| venue | arXiv (preprint, not peer reviewed) |
| id | arXiv:2607.17524 |
| affiliation | University of Southern California |
| upvotes | 6 (Hugging Face Daily Papers, 2026-11-01) |
| links | abs pdf hf |
| abstract | We propose Token-Level Off-Policy Labeling (TOPL), an off-policy training paradigm that reframes post-training as a token-level correctness prediction task. Our key intuition is that by training the model to distinguish good and bad tokens in a response, we naturally guide the model towards generating good tokens, while avoiding the pitfalls that come with directly training the model to generate off-policy tokens. Experiments on document summarization tasks show that TOPL achieves strong out-of-distribution generalization across 11 datasets against a diverse set of sequence-level and token-level baselines. We further demonstrate that TOPL transfers effectively to machine translation, suggesting that its benefits generalize across different faithful generation tasks. Through ablation studies, we confirm that our token-level learning signal is critical to good performance; sequence-level analogues do not confer similar benefits. Finally, we show that TOPL induces interpretable model updates: the LoRA adapters learned through TOPL function as linear classification heads and steering vectors. |
cite
@misc{huang2026token,
title={Token-Level Off-Policy Learning for Faithful Generation Under Distribution Shift},
author={Zitong Huang and Gustavo Lucas Carvalho and Deqing Fu and Robin Jia},
year={2026},
eprint={2607.17524},
archivePrefix={arXiv},
url={https://arxiv.org/abs/2607.17524}
}