| title | ClawGym II: Exploring Black-Box RL on Agent Harness |
|---|---|
| authors | Huatong Song, Fei Bai, Ming Yang, Renyuan Li, Jia Deng, Jujie He, Zhange Zhang, Daixuan Cheng, Yan Xing, Qi Yun, Xuxing Chen, Danyang Li, Feng Chang, Chuan Hao, Ran Tao, Jian Yang, Bryan Dai, Wayne Xin Zhao, Mingjie Tang, Ji-Rong Wen |
| date | 2026-08-17 |
| venue | arXiv (preprint, not peer reviewed) |
| id | arXiv:2608.16798 |
| affiliation | Renmin University of China |
| upvotes | 42 (Hugging Face Daily Papers, 2026-11-01) |
| links | abs pdf hf code |
| abstract | Agent harnesses have substantially improved performance on long-horizon tasks by coordinating agent interactions with the environment. However, reinforcement learning through complex harnesses remains largely unexplored, as scaling such training to long-horizon agent tasks introduces fundamental challenges. In this work, we present a unified black-box RL framework for stable and scalable optimization of general agents through complex harnesses. Concretely, we first build a sandbox-based execution infrastructure that isolates task environments and harnesses within temporary sandboxes for large-scale concurrent rollouts. We then decouple policy optimization from opaque harness execution and place a serving proxy at the model boundary to capture model calls. To reconstruct multi-turn trajectories and improve training efficiency, we organize the captured calls into prefix trees and further adapt both critic-based PPO and critic-free GRPO to optimize over the recovered tree structure. Meanwhile, we maintain training-inference consistency throughout the optimization process. Finally, we introduce mix-harness training, allowing a single model to be jointly optimized by heterogeneous harnesses. With Qwen3-30A3B, black-box RL improves Pass@1 on ClawGym-Bench by 9.98 and 14.81 points through OpenClaw and Claude Code, respectively, while remaining stable over 200-400 optimization steps. Moreover, the framework yields consistent gains on more challenging tasks such as JobBench and OfficeQA. Overall, our framework enables effective, stable, and scalable optimization of general agents through black-box harnesses, supporting unified training across heterogeneous execution systems. |
cite
@misc{song2026clawgym,
title={ClawGym II: Exploring Black-Box RL on Agent Harness},
author={Huatong Song and Fei Bai and Ming Yang and Renyuan Li and Jia Deng and Jujie He and Zhange Zhang and Daixuan Cheng and Yan Xing and Qi Yun and Xuxing Chen and Danyang Li and Feng Chang and Chuan Hao and Ran Tao and Jian Yang and Bryan Dai and Wayne Xin Zhao and Mingjie Tang and Ji-Rong Wen},
year={2026},
eprint={2608.16798},
archivePrefix={arXiv},
url={https://arxiv.org/abs/2608.16798}
}