titleStartupBench: Benchmarking General-Purpose Agents on Market-Validated End-to-End Workflows
authorsLiya Zhu, Xin Ma, Tao Liu, Haodong Wang, Ge Zhang, Jingzhe Ding, Qingshui Gu, Yongjie Zhong, Jinxiang Meng, Yuan Gao, Yunqiu Zhou, Hao Zhu, Jifeng He, Yongzhi Liao, Xinyi Zhang, Chaoxin Li, Yi Zhu, Xi Lin, Duju Zeng, Xiang Gao, Wen Zhang, Yunyang Wang, Duo Wang, Huan Zhou, Zuo Wang, Jin Chen, Kaiyuan Zhang, Chuqian Yu, Tianhao Yu, Longxiang Liu, Jianbo Xue, Huimin Che, Jiahao Wang, Yujia Qin, Jiaheng Liu, Shen Yan, Xiaolong Chang, Wenhao Huang
date2026-08-18
venuearXiv (preprint, not peer reviewed)
idarXiv:2608.17800
affiliationByteDance Seed
upvotes14 (Hugging Face Daily Papers, 2026-11-01)
linksabs pdf hf
abstractRecent advances in Large Language Models(LLMs) and agents have substantially improved the ability of AI systems to execute complex tasks. Yet existing benchmarks largely rely on researcher-selected tasks, leaving uncertain whether such progress extends to the work that real-world users actually demand from AI systems. We introduce StartupBench, an E2E agent benchmark grounded in market-validated AI startup products. Rather than defining tasks from pre-defined assumptions about useful agent capabilities, we systematically study AI products with demonstrated adoption, together with their product workflows and users, to identify real-world tasks for which AI has established practical demand across diverse professional domains. We translate these workflows into complete deliverable-oriented tasks and evaluate them with fine-grained rubrics capturing their complex requirements. Across representative models evaluated under a unified agent harness, even the strongest model successfully completes only approximately 30\% of StartupBench, despite making substantial partial progress on many tasks. Further analysis identifies aspects like complex instruction following and domain-specific expertise as major sources of failure. Our results reveal that many market-validated workflows remain beyond the reliable capabilities of current general-purpose agents, establishing StartupBench as an empirical measure of progress toward E2E completions of real-world user tasks.

cite

@misc{zhu2026startupbench,
  title={StartupBench: Benchmarking General-Purpose Agents on Market-Validated End-to-End Workflows},
  author={Liya Zhu and Xin Ma and Tao Liu and Haodong Wang and Ge Zhang and Jingzhe Ding and Qingshui Gu and Yongjie Zhong and Jinxiang Meng and Yuan Gao and Yunqiu Zhou and Hao Zhu and Jifeng He and Yongzhi Liao and Xinyi Zhang and Chaoxin Li and Yi Zhu and Xi Lin and Duju Zeng and Xiang Gao and Wen Zhang and Yunyang Wang and Duo Wang and Huan Zhou and Zuo Wang and Jin Chen and Kaiyuan Zhang and Chuqian Yu and Tianhao Yu and Longxiang Liu and Jianbo Xue and Huimin Che and Jiahao Wang and Yujia Qin and Jiaheng Liu and Shen Yan and Xiaolong Chang and Wenhao Huang},
  year={2026},
  eprint={2608.17800},
  archivePrefix={arXiv},
  url={https://arxiv.org/abs/2608.17800}
}