| title | Trial by Trace: Benchmarking AI Agent Harnesses for Reliability and Failure Attribution |
|---|---|
| authors | Zhiyong Cao, Hengjie Zhu, Yongxin Wang, Xuchen Li, Jintao Li, Shuxia Liu, Dunqiang Liu, Bo Wang, Kairan Zhang, Cong Lin, JiangDi Yu, Siyuan Liu, Sijia Yao, Yi-Liang Wei, Lingxu Ma, Aijia Wang, Anxin Chen, Zixian Wei, Xiang Ruan, Ruizhi Xu, Ruqian Shi, Baihua Xiao, Sicheng Zhou |
| date | 2026-10-23 |
| venue | arXiv (preprint, not peer reviewed) |
| id | arXiv:2610.32285 |
| affiliation | TraceLite AI |
| upvotes | 66 (Hugging Face Daily Papers, 2026-11-01) |
| links | abs pdf hf code |
| abstract | LLM agents are increasingly used to solve real-world problems, and their effectiveness depends on both the underlying models and the harnesses that support their operation. However, most existing benchmarks focus on model capabilities and overall agent performance, with limited attention to the evaluation of agent harnesses. To address this gap, we introduce TraceTrial-Bench, which organizes harness reliability into seven components covering context, execution, governance, lifecycle, observability, tools, and verification. We construct 95 expert-level, mechanism-oriented tasks with executable or state-based ground truth to support semantically equivalent comparisons while preserving the native behavior of each harness. We design a Trajectory Tracing (TT) module to capture agent execution traces and introduce a two-layer evaluation framework that separately assesses agent performance and harness reliability. In addition, we design a Multi-source Evidence Acquisition (MEA) module to provide evidence from the agent context, harness, and execution environment for LLM-based evaluation, and develop a Failure Attribution (FA) module that uses explicit mechanism chains to support attribution at the run level. Across 4 harnesses, 9 models, and 95 tasks, our evaluation of 3,420 runs yields macro-averaged harness scores ranging from 59.8% to 72.0%, while 31.1% of runs achieve task success despite failing at least one harness criterion. TraceTrial-Bench provides an auditable and reproducible foundation for comparing agent harnesses, diagnosing their failures, and studying how harness design interacts with model behavior. |
cite
@misc{cao2026trial,
title={Trial by Trace: Benchmarking AI Agent Harnesses for Reliability and Failure Attribution},
author={Zhiyong Cao and Hengjie Zhu and Yongxin Wang and Xuchen Li and Jintao Li and Shuxia Liu and Dunqiang Liu and Bo Wang and Kairan Zhang and Cong Lin and JiangDi Yu and Siyuan Liu and Sijia Yao and Yi-Liang Wei and Lingxu Ma and Aijia Wang and Anxin Chen and Zixian Wei and Xiang Ruan and Ruizhi Xu and Ruqian Shi and Baihua Xiao and Sicheng Zhou},
year={2026},
eprint={2610.32285},
archivePrefix={arXiv},
url={https://arxiv.org/abs/2610.32285}
}