titleFrontierCS 2: A Benchmark for Computer Science R&D
authorsQiuyang Mang, Wenhao Chai, Runyuan He, Dongruixuan Li, Zihan Zheng, Yujun He, Xuanyi Zhou, Hao Wang, Siddhant Ray, Bo Peng, Ruyi Ji, Shuo Yang, Haocheng Xi, Boru Chen, Xinyang David Han, Melissa Pan, Mayank Mishra, Boxi Yu, Sijia Liu, Kerem Akillioglu, Yi Pan, Zirui Wang, Mert Cemri, Diogo Antunes, Yichuan Wang, Aiwei Ivy Zhang, Zhenyang Chen, Qizheng Zhang, Kaiyuan Liu, Wenyuan Huang, Sherry Ruan, Shu Liu, Dacheng Li, Zerui Cheng, Zeyu Shen, Shang Zhou, Hanfei Chen, Yuzhong Zhang, Bohan Lyu, Qu Ao, Han Zheng, Chonghe Jiang, Jingzhuo Hu, Xiaokun Chen, Kunjun Li, Zhiqing Zhong, Haoye Tian, Tong Zhao, Yunzhong He, Hanchen Li, Ofir Press, Jiantao Jiao, Kunle Olukotun, Karthik R. Narasimhan, Jingbo Shang, Koushik Sen, Matei Zaharia, Alex Dimakis, Ion Stoica, Joseph E. Gonzalez, Alvin Cheung
date2026-10-21
venuearXiv (preprint, not peer reviewed)
idarXiv:2610.29932
upvotes4 (Hugging Face Daily Papers, 2026-11-01)
linksabs pdf hf code
abstractCan LLM agents make contributions to frontier computer science R&D? We introduce FrontierCS 2, a benchmark of 48 open-ended tasks. We take a paper and the code its authors released and delete the paper's contribution. The deleted code becomes the human reference that sets the bar. Earlier open-ended benchmarks give each agent a problem statement and grade one program after the run. FrontierCS 2 places the agent inside a research environment. The agent inherits a working repository and may submit a patch at any point. For each patch, a judge returns a development-suite score but keeps the evaluator, the held-out final suite and the reference private, and grades the final submission on that suite against the reference. The tasks span 13 domains, from machine-learning systems and high-performance computing to databases, security and robotics. Beyond these 48 tasks, FrontierCS 2 keeps a private set of more than 100 tasks, to be announced in the future. We evaluate 12 agents. The best one is Claude Opus 5.5, passing 58% of its runs. No run passes on 16 tasks. A higher score is not the same as a pass: most runs that beat the reference on the combined final score still fall short on some metric or workload. On a shared capability scale for agents and tasks, the median task is roughly as difficult as what the strongest agent can handle.

cite

@misc{mang2026frontiercs,
  title={FrontierCS 2: A Benchmark for Computer Science R&D},
  author={Qiuyang Mang and Wenhao Chai and Runyuan He and Dongruixuan Li and Zihan Zheng and Yujun He and Xuanyi Zhou and Hao Wang and Siddhant Ray and Bo Peng and Ruyi Ji and Shuo Yang and Haocheng Xi and Boru Chen and Xinyang David Han and Melissa Pan and Mayank Mishra and Boxi Yu and Sijia Liu and Kerem Akillioglu and Yi Pan and Zirui Wang and Mert Cemri and Diogo Antunes and Yichuan Wang and Aiwei Ivy Zhang and Zhenyang Chen and Qizheng Zhang and Kaiyuan Liu and Wenyuan Huang and Sherry Ruan and Shu Liu and Dacheng Li and Zerui Cheng and Zeyu Shen and Shang Zhou and Hanfei Chen and Yuzhong Zhang and Bohan Lyu and Qu Ao and Han Zheng and Chonghe Jiang and Jingzhuo Hu and Xiaokun Chen and Kunjun Li and Zhiqing Zhong and Haoye Tian and Tong Zhao and Yunzhong He and Hanchen Li and Ofir Press and Jiantao Jiao and Kunle Olukotun and Karthik R. Narasimhan and Jingbo Shang and Koushik Sen and Matei Zaharia and Alex Dimakis and Ion Stoica and Joseph E. Gonzalez and Alvin Cheung},
  year={2026},
  eprint={2610.29932},
  archivePrefix={arXiv},
  url={https://arxiv.org/abs/2610.29932}
}