| title | LLM-as-a-Coach: Experiential Learning for Non-Verifiable Tasks |
|---|---|
| authors | Tianzhu Ye, Li Dong, Guanheng Chen, He Zhu, Xun Wu, Shaohan Huang, Furu Wei |
| date | 2026-07-20 |
| venue | arXiv (preprint, not peer reviewed) |
| id | arXiv:2607.18110 |
| affiliation | Microsoft Research |
| upvotes | 16 (Hugging Face Daily Papers, 2026-11-01) |
| links | abs pdf hf |
| abstract | Reinforcement learning (RL) on open-ended tasks compresses an LLM's rubric-based evaluation into a scalar reward, discarding rich textual feedback and conflating responses with distinct quality profiles. We propose Experiential Learning (EL), which repurposes the feedback model from an LLM-as-a-Judge into an LLM-as-a-Coach. The coach distills its assessment of each on-policy response into transferable experiential knowledge, which conditions a teacher model and is internalized by the policy through on-policy context distillation. Compared with scalar rewards, this higher-bandwidth feedback channel provides dense supervision and preserves fine-grained preferences among high-quality responses. Across two policy families, with feedback from the policy itself or a proprietary model, EL consistently outperforms rubric-based RL on held-out and unseen open-ended tasks. Notably, EL generalizes better beyond the training distribution, and mitigates reward hacking. These findings establish experiential knowledge as a richer and more generalizable learning signal for post-training on non-verifiable tasks. |
cite
@misc{ye2026llm,
title={LLM-as-a-Coach: Experiential Learning for Non-Verifiable Tasks},
author={Tianzhu Ye and Li Dong and Guanheng Chen and He Zhu and Xun Wu and Shaohan Huang and Furu Wei},
year={2026},
eprint={2607.18110},
archivePrefix={arXiv},
url={https://arxiv.org/abs/2607.18110}
}