| title | Training Foundation Models with a Single Algorithm |
|---|---|
| authors | Yuda Song, Alvin Zhang, Fahim Tajwar, Daman Arora, Jeff Schneider, Daniel Khashabi, Ruslan Salakhutdinov, J. Andrew Bagnell, Aarti Singh, Andrea Zanette |
| date | 2026-10-29 |
| venue | arXiv (preprint, not peer reviewed) |
| id | arXiv:2610.40564 |
| affiliation | Carnegie Mellon University |
| upvotes | 8 (Hugging Face Daily Papers, 2026-11-01) |
| links | abs pdf hf code |
| abstract | Foundation models are typically trained in stages: pretraining, midtraining, supervised fine-tuning (SFT), and reinforcement learning (RL), with changes in both data and learning algorithms. We show that this algorithmic separation is not fundamental. By deriving suitable RL objectives as maximum-likelihood fitting to model-generated target distributions, we place all of these stages under a single likelihood-based training algorithm, unified training. This perspective turns the foundation-model training lifecycle into a problem of choosing which data to train on, in what proportions, and when, with conventional staged training as a special case. We instantiate this framework in a unified training system that mixes static data and RL generations within a single trainer. Across controlled and real-world experiments, unified training improves the accuracy-compute tradeoff and task coverage while preserving broader capabilities, and provides a stronger starting point for continual learning. This work lays the groundwork for a unified science of foundation-model training, connecting mathematical principles, training infrastructure, and the development of model capabilities. |
cite
@misc{song2026training,
title={Training Foundation Models with a Single Algorithm},
author={Yuda Song and Alvin Zhang and Fahim Tajwar and Daman Arora and Jeff Schneider and Daniel Khashabi and Ruslan Salakhutdinov and J. Andrew Bagnell and Aarti Singh and Andrea Zanette},
year={2026},
eprint={2610.40564},
archivePrefix={arXiv},
url={https://arxiv.org/abs/2610.40564}
}