| title | Trust Region Q Adjoint Matching |
|---|---|
| authors | Yonghoon Dong, Kyungmin Lee, Changyeon Kim, Jaehyuk Kim, Jinwoo Shin |
| date | 2026-05-26 |
| venue | arXiv (preprint, not peer reviewed) |
| id | arXiv:2605.27079 |
| affiliation | KAIST AI |
| upvotes | 24 (Hugging Face Daily Papers, 2026-11-01) |
| links | abs pdf hf code |
| abstract | Off-policy reinforcement learning of pretrained flow policies remains challenging due to the instability of optimization arising from the multi-step sampling process. Recently, Q-learning with Adjoint Matching (QAM) addressed this issue by reformulating into a memoryless stochastic optimal control (SOC) problem with a learned critic. However, QAM inherits a fundamental fragility of critic-guided improvement: small critic errors are amplified when critics are ill-conditioned, often leading to model collapse. This paper introduces Trust Region Q-Adjoint Matching (TRQAM), a stable off-policy fine-tuning algorithm that adaptively controls the path-space KL with pretrained flow policies through projected dual descent. Specifically, we optimize the trust-region parameter λ in SOC dynamics, and theoretically show that the path-space KL can be represented by a closed-form function of λ. As a result, our method can precisely control the exact deviation from pretrained flow policies, achieving stable off-policy RL. Through experiments on 50 OGBench tasks, TRQAM consistently outperforms prior arts in both offline RL and offline-to-online RL. In particular, TRQAM achieves an overall success rate of 68% in offline RL, substantially improves the strongest baseline at 46%. |
cite
@misc{dong2026trust,
title={Trust Region Q Adjoint Matching},
author={Yonghoon Dong and Kyungmin Lee and Changyeon Kim and Jaehyuk Kim and Jinwoo Shin},
year={2026},
eprint={2605.27079},
archivePrefix={arXiv},
url={https://arxiv.org/abs/2605.27079}
}