titleExperience-Handoff Reinforcement Learning for Recursive Self-Improvement
authorsDoyoung Kim, Byeongguk Jeon, Kailin Zhang, Sherry Yang
date2026-11-12
venuearXiv (preprint, not peer reviewed)
idarXiv:2611.16409
affiliationKAIST AI
upvotes4 (Hugging Face Daily Papers, 2026-11-15)
linksabs pdf hf code
cite
relatedOmniShowcase: Omnidirectional Subject-Consistent Video Generation from Multi-View Images
Look Only When Needed: Certificate-Guided Visual Routing for Efficient Diffusion VLMs
Language Models Can Control Their Own Attention
OmniRetrieval: Unified Retrieval across Heterogeneous Knowledge Sources
GRACE: Generation-aware latent compression for efficient video generation
abstractIt is common to train Large Language Models (LLMs) for agentic tasks with Reinforcement Learning (RL) from scalar rewards. However, this throws away the fine-grained information about the environment that the agent experienced during the trial (e.g., error logs). To utilize this rich information, recent works wrap the agent in a harness that refines its own inputs through reflection or compaction and hands past experience to future rollouts, but the construction of the handed-off artifacts (e.g., a playbook) is itself not learned. We want the best of both worlds: gradient updates from a scalar reward, while also utilizing environment information to hand off experience. As a concrete testbed, we fix the Agentic Context Engineering () harness and train one model to act, reflect, and curate task-local playbooks. Our method, , evaluates candidate experience-handoff artifacts by their downstream effect on future actor rollouts, and co-optimizes the role-specific behaviors (action generator, insight reflector, and playbook curator) under a fixed scaffold. Across agentic benchmarks, achieves higher held-out performance than baseline methods. The results suggest that agent training should optimize not only the actions in the current rollout, but also the model-written experience that future rollouts inherit.

bibtex

@misc{kim2026experience,
  title={Experience-Handoff Reinforcement Learning for Recursive Self-Improvement},
  author={Doyoung Kim and Byeongguk Jeon and Kailin Zhang and Sherry Yang},
  year={2026},
  eprint={2611.16409},
  archivePrefix={arXiv},
  url={https://arxiv.org/abs/2611.16409}
}