titleROSS: Relearning from Self-Generated Rollouts through Selective Supervision
authorsZhiwei Zhang, Huayu Deng, Fei Zhao, Jiayan Fu, Bin Liang, Kam-Fai Wong, Mu Chuan
date2026-09-28
venuearXiv (preprint, not peer reviewed)
idarXiv:2609.35954
affiliationAllSpark Research
upvotes34 (Hugging Face Daily Papers, 2026-11-01)
linksabs pdf hf code
abstractLarge language model post-training generates self-generated rollouts through reinforcement learning and on-policy distillation, yet this experience is often treated as stale once the policy advances. Historical rollouts can remain compatible with a later policy while preserving behaviors that the policy no longer expresses reliably. However, they may also contain mistakes, abandoned attempts, and redundant actions that should not be imitated, motivating finer-grained selective supervision. We introduce ROSS (Relearning from Self-Generated Rollouts through Selective Supervision), which preserves the full historical trajectory as context while applying loss only to selected model-generated continuations. Across domain-specific reinforcement learning, multi-teacher on-policy distillation, and agentic reinforcement learning, ROSS consistently improves upstream checkpoints and outperforms baselines across mathematics, code generation, instruction following, and software engineering. On Qwen3.6-35B-A3B, ROSS improves the six-benchmark MOPD average from 58.40% to 62.20% and SWE-bench Verified from 64.20% to 68.40%. These results show that self-rollout training leaves behind reusable behavioral experience that can yield further gains through offline supervised fine-tuning (SFT), without additional policy rollouts.

cite

@misc{zhang2026ross,
  title={ROSS: Relearning from Self-Generated Rollouts through Selective Supervision},
  author={Zhiwei Zhang and Huayu Deng and Fei Zhao and Jiayan Fu and Bin Liang and Kam-Fai Wong and Mu Chuan},
  year={2026},
  eprint={2609.35954},
  archivePrefix={arXiv},
  url={https://arxiv.org/abs/2609.35954}
}