| title | CLI-Universe: Towards Verifiable Task Synthesis Engine for Terminal Agents |
|---|---|
| authors | Zhanbo Hua, Yifan Yao, Weihao Xie, Yongchi Zhao, Minghao Liu, Ruizhi Qiu, Zhewei Huang, Zun Wang, Yiyan Ji, Yunhai Ye, Letian Zhu, Xinping Lei, Han Li, Zhiyuan Ma, Zili Wang, Zhaoxiang Zhang, Jiaheng Liu |
| date | 2026-06-22 |
| venue | arXiv (preprint, not peer reviewed) |
| id | arXiv:2606.22883 |
| affiliation | NJU-LINK Lab |
| upvotes | 38 (Hugging Face Daily Papers, 2026-11-01) |
| links | abs pdf hf |
| abstract | While recent LLM-based terminal agents have demonstrated promising capabilities, the scarcity of high-quality, executable training data remains a critical bottleneck. Existing synthesis pipelines typically scale by retrofitting surface-level artifacts into tasks, frequently yielding ambiguous instructions, shallow execution paths, and brittle tests that provide weak learning signals. To overcome this, we introduce CLI-Universe, a principled synthesis engine that constructs terminal-agent tasks. CLI-Universe generates candidate tasks by sampling combinations across a multi-dimensional capability taxonomy (domain, skill type, capability, and engineering pillar), then grounds each candidate through evidence-guided deep research over real-world technical materials. To ensure rigorous supervision, validated blueprints are instantiated into Dockerized environments and subjected to a multi-stage executable verification pipeline featuring rubric-gated test construction, hint-conditional filtering, and strict fail-to-pass checking. Across the full pipeline, from candidate generation to verification, approximately two-thirds of candidates are discarded, retaining only those that are genuine, verifiable, and non-trivially challenging. To validate our framework, we instantiate a highly distilled dataset of 6,000 trajectories called CLI-Universe-6K. Remarkably, fine-tuning Qwen3-32B on CLI-Universe-6K achieves 33.4% on Terminal-Bench 2.0. This sets a new state-of-the-art for models trained on open-source data at or below 32B parameters, and outperforms several models an order of magnitude larger, demonstrating the profound data efficiency of structured, high-fidelity synthesis. |
cite
@misc{hua2026cli,
title={CLI-Universe: Towards Verifiable Task Synthesis Engine for Terminal Agents},
author={Zhanbo Hua and Yifan Yao and Weihao Xie and Yongchi Zhao and Minghao Liu and Ruizhi Qiu and Zhewei Huang and Zun Wang and Yiyan Ji and Yunhai Ye and Letian Zhu and Xinping Lei and Han Li and Zhiyuan Ma and Zili Wang and Zhaoxiang Zhang and Jiaheng Liu},
year={2026},
eprint={2606.22883},
archivePrefix={arXiv},
url={https://arxiv.org/abs/2606.22883}
}