| title | AstraFlow: Dataflow-Oriented Reinforcement Learning for Agentic LLMs |
|---|---|
| authors | Haizhong Zheng, Yizhuo Di, Jiahui Wang, Shuowei Jin, Xueshen Liu, Yongji Wu, Z. Morley Mao, Ion Stoica, Jiawei Zhao, Beidi Chen |
| date | 2026-05-15 |
| venue | arXiv (preprint, not peer reviewed) |
| id | arXiv:2605.15565 |
| affiliation | Carnegie Mellon University |
| upvotes | 15 (Hugging Face Daily Papers, 2026-11-01) |
| links | abs pdf hf code |
| abstract | Reinforcement learning (RL) is increasingly used to improve the reasoning, coding, and tool-use capabilities of large language models, but agentic RL remains prohibitively expensive. Scaling RL to agentic LLMs requires supporting complex workloads, including multi-policy collaborative training, while efficiently using elastic, heterogeneous, and cross-region compute resources. Existing LLM RL systems support some of these capabilities, but each new extension often requires dedicated system engineering. This burden arises from trainer-centered control architectures and the lack of principled abstractions for RL system components. To address these limitations, we propose AstraFlow, a dataflow-oriented RL system that replaces conventional trainer-centered control with principled component abstractions. In AstraFlow, rollout services, dataflow management, and training are decoupled into autonomous components, enabling the system to natively support complex multi-policy agentic RL workloads and efficiently exploit diverse compute resources. We evaluate AstraFlow across math, code, search, and AgentBench workloads, showing that the same system supports multi-policy training, elastic scaling, heterogeneous cross-region execution, and composable data algorithms without system-level code changes. In multi-policy collaborative training, AstraFlow achieves comparable or better accuracy than existing RL systems while speeding up training time by 2.7x. |
cite
@misc{zheng2026astraflow,
title={AstraFlow: Dataflow-Oriented Reinforcement Learning for Agentic LLMs},
author={Haizhong Zheng and Yizhuo Di and Jiahui Wang and Shuowei Jin and Xueshen Liu and Yongji Wu and Z. Morley Mao and Ion Stoica and Jiawei Zhao and Beidi Chen},
year={2026},
eprint={2605.15565},
archivePrefix={arXiv},
url={https://arxiv.org/abs/2605.15565}
}