| title | DiffusionOPD: A Unified Perspective of On-Policy Distillation in Diffusion Models |
|---|---|
| authors | Quanhao Li, Junqiu Yu, Kaixun Jiang, Yujie Wei, Zhen Xing, Pandeng Li, Ruihang Chu, Shiwei Zhang, Yu Liu, Zuxuan Wu |
| date | 2026-05-14 |
| venue | arXiv (preprint, not peer reviewed) |
| id | arXiv:2605.15055 |
| upvotes | 17 (Hugging Face Daily Papers, 2026-11-01) |
| links | abs pdf hf code |
| abstract | Reinforcement learning has emerged as a powerful tool for improving diffusion-based text-to-image models, but existing methods are largely limited to single-task optimization. Extending RL to multiple tasks is challenging: joint optimization suffers from cross-task interference and imbalance, while cascade RL is cumbersome and prone to catastrophic forgetting. We propose DiffusionOPD, a new multi-task training paradigm for diffusion models based on Online Policy Distillation (OPD). DiffusionOPD first trains task-specific teachers independently, then distills their capabilities into a unified student along the student own rollout trajectories. This decouples single-task exploration from multi-task integration and avoids the optimization burden of solving all tasks jointly from scratch. Theoretically, we lift the OPD framework from discrete tokens to continuous-state Markov processes, deriving a closed-form per-step KL objective that unifies both stochastic SDE and deterministic ODE refinement via mean-matching. We formally and empirically demonstrate that this analytic gradient provides lower variance and better generality compared to conventional PPO-style policy gradients. Extensive experiments show that DiffusionOPD consistently surpasses both multi-reward RL and cascade RL baselines in training efficiency and final performance, while achieving state-of-the-art results on all evaluated benchmarks. |
cite
@misc{li2026diffusionopd,
title={DiffusionOPD: A Unified Perspective of On-Policy Distillation in Diffusion Models},
author={Quanhao Li and Junqiu Yu and Kaixun Jiang and Yujie Wei and Zhen Xing and Pandeng Li and Ruihang Chu and Shiwei Zhang and Yu Liu and Zuxuan Wu},
year={2026},
eprint={2605.15055},
archivePrefix={arXiv},
url={https://arxiv.org/abs/2605.15055}
}