| title | Reinforcement Learning from Rich Feedback with Distributional DAgger |
|---|---|
| authors | Rishabh Agrawal, Jacob Fein-Ashley, Paria Rashidinejad |
| date | 2026-06-03 |
| venue | arXiv (preprint, not peer reviewed) |
| id | arXiv:2606.05152 |
| affiliation | University of Southern California |
| upvotes | 4 (Hugging Face Daily Papers, 2026-11-01) |
| links | abs pdf hf code |
| abstract | Reasoning models have advanced rapidly, but the dominant reinforcement learning from verifiable rewards (RLVR) recipe remains surprisingly narrow: sample many responses and reward each with a single bit indicating whether the final answer is correct. Yet many settings provide rich feedback, including execution traces, tool outputs, expert corrections, and model self-evaluations. We study how to use such feedback through a distributional variant of the classic imitation learning algorithm DAgger, where the learner has local access to an expert distribution on states visited by the current policy. This yields a simple forward cross-entropy objective that admits a blackbox expert and whose sequence-level gradient {conduct rich credit assignment by propagating} future expert-student disagreement back to earlier decisions. We show that prior RL with self-distillation objectives based on reverse KL or Jensen-Shannon fail to guarantee monotonic policy improvement: even when the expert has higher reward, their updates may increase probability on worse actions. In contrast, we show that forward cross-entropy admits monotonic policy improvement and enjoys guarantees on regret. We further show that our objective optimizes a lower bound on teacher-weighted likelihood of success, leading to improved Pass@N. Empirically, our approach, DistIL, improves over RLVR and RL with self-distillation baselines across a variety of domains: scientific reasoning, coding, and solving hard mathematical problems. |
cite
@misc{agrawal2026reinforcement,
title={Reinforcement Learning from Rich Feedback with Distributional DAgger},
author={Rishabh Agrawal and Jacob Fein-Ashley and Paria Rashidinejad},
year={2026},
eprint={2606.05152},
archivePrefix={arXiv},
url={https://arxiv.org/abs/2606.05152}
}