titleFewStepRL: Fine-tuning Few-step Diffusion Models with Reinforcement Learning
authorsHaitao Lin, Minsi Ren, Yang Li, Xingtong Ge, Zheyuan Hu, Peiyan Hu, Tailin Wu
date2026-11-05
venuearXiv (preprint, not peer reviewed)
idarXiv:2611.08240
affiliationTencent Hunyuan
upvotes9 (Hugging Face Daily Papers, 2026-11-12)
linksabs pdf hf code
cite
relatedAuK Technical Report: An Open-Source Foundational Model for Speech Generation and Editing
Hy-Nomas: Towards Free-form Interactive World Generation
Harness Handbook: Making Evolving Agent Harnesses Readable,Navigable, and Editable
Hierarchical Sparse Attention Done Right: Toward Infinite Context Modeling
Recursive Synthesis for Long-Horizon Terminal Tasks
abstractReinforcement learning (RL) has become a standard post-training method for diffusion models, but most RL objectives roll out the current policy at every iteration, and with a many-step sampler these rollouts typically dominate the cost of training. Few-step models cut this cost, yet they model an average velocity over an interval, whereas existing RL objectives are built for an instantaneous velocity with a many-step sampler. We adapt RL objectives to few-step models systematically in three phases, namely a transformation, a decomposition and an adaptation. The transformation to x-space expresses each objective we consider through a common policy prediction, at the clean end by default and at an explicitly marked level for transition or flow-map objectives, so that objectives stated over densities, velocities or rewards become comparable. The decomposition writes the gradient of each of these objectives as the sum of a self-regression toward a reference and a reward augmentation, so that a method reduces, at the gradient level, to a choice of anchors and weights. The adaptation computes each anchor from a few-step model at the end of its transport or along its tangent, with the instantaneous velocity recovered from the network itself or from an auxiliary network. This yields few-step versions of FlowGRPO, AWM, DiffusionNFT, DiffusionOPSD and ReFL in one modular codebase. In our experiments, reward-gradient-based methods are the most effective when the reward is differentiable and its gradient is affordable; otherwise DiffusionNFT-style methods generally reach high rewards in relatively few iterations despite a sometimes higher per-iteration cost. Extra inference steps show no systematic aggregate benefit in our experiments, although selected outputs visibly improve with two to four times as many sampling steps as in training.

bibtex

@misc{lin2026fewsteprl,
  title={FewStepRL: Fine-tuning Few-step Diffusion Models with Reinforcement Learning},
  author={Haitao Lin and Minsi Ren and Yang Li and Xingtong Ge and Zheyuan Hu and Peiyan Hu and Tailin Wu},
  year={2026},
  eprint={2611.08240},
  archivePrefix={arXiv},
  url={https://arxiv.org/abs/2611.08240}
}