| title | StepPO: Step-Aligned Policy Optimization for Agentic Reinforcement Learning |
|---|---|
| authors | Daoyu Wang, Qingchuan Li, Mingyue Cheng, Jie Ouyang, Shuo Yu, Qi Liu, Enhong Chen |
| date | 2026-06-05 |
| venue | arXiv (preprint, not peer reviewed) |
| id | arXiv:2604.18401 |
| upvotes | 6 (Hugging Face Daily Papers, 2026-11-01) |
| links | abs pdf hf code |
| abstract | Agentic reinforcement learning (RL) is emerging as a critical post-training paradigm for improving LLM agent capabilities. Existing RL algorithms for LLMs largely follow the token-centric paradigm as in RLHF and RLVR, where tokens serve as the basic units for modeling and optimization. However, this paradigm introduces a granularity mismatch in agentic RL, as it optimizes token-level predictions while LLM agents make step-level decisions through cycles of environmental observations and actions. To bridge this gap, we propose StepPO, a step-centric paradigm for agentic RL via step-aligned policy optimization. Specifically, we reformulate agentic RL from a token-level Markov Decision Process (MDP) into a step-level MDP, where interaction steps serve as the basic trajectory representations. We further propose step-level credit assignment to align policy optimization with the natural granularity of agent decisions. Together, StepPO optimizes agent policies at the step level for multi-turn agent-environment interaction. Experiments across multi-hop QA, academic paper search, and text-world action tasks show that StepPO consistently outperforms various RL algorithms. Further analyses provide insights into how step-centric paradigm improves agent training. We hope this step-centric paradigm offers a useful lens for understanding agent behavior and a practical path for training more capable LLM agents. |
cite
@misc{wang2026steppo,
title={StepPO: Step-Aligned Policy Optimization for Agentic Reinforcement Learning},
author={Daoyu Wang and Qingchuan Li and Mingyue Cheng and Jie Ouyang and Shuo Yu and Qi Liu and Enhong Chen},
year={2026},
eprint={2604.18401},
archivePrefix={arXiv},
url={https://arxiv.org/abs/2604.18401}
}