titleDVAO: Dynamic Variance-adaptive Advantage Optimization for Multi-reward Reinforcement Learning
authorsGuochao Jiang, Jingyi Song, Guofeng Quan, Chuzhan Hao, Guohua Liu, Yuewei Zhang
date2026-05-25
venuearXiv (preprint, not peer reviewed)
idarXiv:2605.25604
upvotes39 (Hugging Face Daily Papers, 2026-11-01)
linksabs pdf hf
abstractReinforcement Learning has become a standard paradigm for aligning Large Language Models with human intent and task requirements. While Group Relative Policy Optimization offers an efficient, value-model-free alternative to Proximal Policy Optimization, adapting it to real-world multi-reward settings remains challenging. Standard scalarization practices, such as Reward Combination and Advantage Combination, suffer from significant drawbacks: Reward Combination frequently generates advantages with excessively large squared magnitudes that lead to training instability, while Advantage Combination relies on static hyperparameters and ignores cross-objective correlations. To address these limitations, we propose Dynamic Variance-adaptive Advantage Optimization (DVAO), which dynamically adjusts combination weights based on the empirical reward variance of each objective within a rollout group, effectively up-weighting objectives with a stronger learning signal while suppressing noisy ones. We mathematically prove that DVAO maintains bounded advantage magnitudes for stable training and introduces a self-adaptive cross-objective regularization mechanism. Extensive experiments on mathematical reasoning and tool-use benchmarks using Qwen3 and Qwen2.5 models demonstrate that DVAO significantly outperforms baseline methods, achieving a superior multi-objective Pareto frontier and robust training stability.

cite

@misc{jiang2026dvao,
  title={DVAO: Dynamic Variance-adaptive Advantage Optimization for Multi-reward Reinforcement Learning},
  author={Guochao Jiang and Jingyi Song and Guofeng Quan and Chuzhan Hao and Guohua Liu and Yuewei Zhang},
  year={2026},
  eprint={2605.25604},
  archivePrefix={arXiv},
  url={https://arxiv.org/abs/2605.25604}
}