titleRL Post-Training in Diffusion Transformers Is Activation Steering
authorsTuna Han Salih Meral, Kaan Oktay, Alec Helbling, Atakan Kara, Duen Horng Chau, Pinar Yanardag
date2026-11-19
venuearXiv (preprint, not peer reviewed)
idarXiv:2611.26627
upvotes14 (Hugging Face Daily Papers, 2026-11-21)
linksabs pdf hf
cite
abstractReinforcement learning (RL) post-training substantially improves the visual quality, text rendering, and compositional accuracy of diffusion transformers. Because capabilities such as rendering requested text or composing multiple objects are inherently prompt-dependent, a natural hypothesis is that improving them requires learning complex, input-dependent transformations. To isolate how much of these gains an input-independent shift explains, we probe each RL-tuned model with a deliberately simple parameterization: a fixed set of steering vectors added to the block outputs of the frozen base model. On SD3.5-M, these vectors hold 65x fewer parameters than the RL-tuned LoRA (0.29M against 18.8M). We fit them to match the velocity predictions of public RL-tuned checkpoints, with no reward in the objective. Although the vectors are identical for every prompt, token, and sampling seed, they recover 90.0-96.3% of the reward gains of six RL-tuned image models, and 89.8% and 111.9% on two RL-tuned video models. Initialized at zero and trained with online RL, the vectors alone exceed the reward gains of weight-based RL on text rendering (OCR) and human preference (PickScore) on SD3.5-M. A constant shift thus suffices for most of the gain of policy-gradient RL post-training: without new input-dependent transformations, it steers the pretrained generator toward what it can already compute.

bibtex

@misc{meral2026rl,
  title={RL Post-Training in Diffusion Transformers Is Activation Steering},
  author={Tuna Han Salih Meral and Kaan Oktay and Alec Helbling and Atakan Kara and Duen Horng Chau and Pinar Yanardag},
  year={2026},
  eprint={2611.26627},
  archivePrefix={arXiv},
  url={https://arxiv.org/abs/2611.26627}
}