| title | Language Models Learn to Improve Language Models |
|---|---|
| authors | Minhua Lin, Zijun Wang, Zewen Liu, Zhaotian Weng, Bing He, Zhan Shi, Benoit Dumoulin, Xiang Zhang, Suhang Wang, Hanqing Lu |
| date | 2026-10-16 |
| venue | arXiv (preprint, not peer reviewed) |
| id | arXiv:2610.23344 |
| upvotes | 20 (Hugging Face Daily Papers, 2026-11-01) |
| links | abs pdf hf |
| primer | World models |
| abstract | Large language models (LLMs) are increasingly used to automate AI research, including language-model (LM) development, offering a potential path toward recursive self-improvement. However, turning experimental feedback into a learned ability to propose effective improvements across models faces two challenges. First, training such an ability with real feedback is expensive: reinforcement learning (RL) requires evaluating tens of thousands of proposals, each potentially requiring a full model-training run. Even at ten GPU-hours per run, reward collection alone would consume hundreds of thousands of GPU-hours, motivating a cheaper surrogate. Second, surrogate feedback can become unreliable as policy optimization favors overestimated proposals when the proposal distribution shifts. To address these challenges, we propose LM2LM, which trains LLMs with RL to generate effective LM improvement proposals. To provide scalable feedback, we use a research world model (RWM), a frozen LLM that predicts proposal gains from real experimental records provided in context, as the reward model for RL. To maintain feedback reliability, we periodically execute current-policy proposals ranked highly by the RWM and add their measured outcomes to its context. Across eight autoregressive and diffusion LM development environments, real-execution evaluations on held-out tasks show that LM2LM enables Qwen3.8-27B to outperform GPT-6-sol, even when GPT-6-sol is given 300 real experimental records in context, while requiring 33% lower data-generation cost. LM2LM also generalizes to a model family unseen during training. |
cite
@misc{lin2026language,
title={Language Models Learn to Improve Language Models},
author={Minhua Lin and Zijun Wang and Zewen Liu and Zhaotian Weng and Bing He and Zhan Shi and Benoit Dumoulin and Xiang Zhang and Suhang Wang and Hanqing Lu},
year={2026},
eprint={2610.23344},
archivePrefix={arXiv},
url={https://arxiv.org/abs/2610.23344}
}