| title | When Tools Fail: Benchmarking Dynamic Replanning and Anomaly Recovery in LLM Agents |
|---|---|
| authors | Dongsheng Zhu, Xuchen Ma, Yucheng Shen, Xiang Li, Yukun Zhao, Shuaiqiang Wang, Lingyong Yan, Dawei Yin |
| date | 2026-06-04 |
| venue | arXiv (preprint, not peer reviewed) |
| id | arXiv:2606.05806 |
| affiliation | BAIDU |
| upvotes | 22 (Hugging Face Daily Papers, 2026-11-01) |
| links | abs pdf hf code |
| abstract | Existing benchmarks evaluate Tool-Integrated Reasoning (TIR) in LLMs on idealized ''happy paths'', largely overlooking real-world tool failures. We introduce ToolMaze, a benchmark for dynamic path discovery and error recovery in TIR agents. To separate systematic replanning from blind trial-and-error, ToolMaze adopts a two-dimensional design: DAG-based topological complexity and a 2 times 2 taxonomy of tool perturbations (explicit/implicit, transient/permanent). Evaluations show that perturbations degrade performance across nearly all models, with the sharpest drops under implicit semantic failures. Driven by systemic over-trust in corrupted outputs, Perturbation Recovery Rate (PRR) plummets by around 37\% in these scenarios, while complex topologies trap agents in futile trial-and-error loops. Crucially, agentic fault-tolerance improves with model scale 3.66times slower than basic task execution, highlighting dynamic replanning as a distinct bottleneck unaddressed by model scaling or prompting. Data and code are available at https://github.com/Zhudongsheng75/ToolMaze. |
cite
@misc{zhu2026when,
title={When Tools Fail: Benchmarking Dynamic Replanning and Anomaly Recovery in LLM Agents},
author={Dongsheng Zhu and Xuchen Ma and Yucheng Shen and Xiang Li and Yukun Zhao and Shuaiqiang Wang and Lingyong Yan and Dawei Yin},
year={2026},
eprint={2606.05806},
archivePrefix={arXiv},
url={https://arxiv.org/abs/2606.05806}
}