| title | Running the Gauntlet: Re-evaluating the Capabilities of Agents Beyond Familiar Environments |
|---|---|
| authors | Mykola Vysotskyi, Runqi Lin, Grzegorz Biziel, Michal Zakrzewski, Sebastian Montagna, Damian Rynczak, Shreyansh Padarha, Kumail Alhamoud, Zihao Fu, William Lugoloobi, Kai Rawal, Hanna Yershova, Xander Davies, Taras Rumezhak, Guohao Li, Fazl Barez, Baoyuan Wu, Arkadiusz Drohomirecki, Yarin Gal, Chris Russell, Christopher Summerfield, Adam Mahdi, Volodymyr Karpiv, Philip Torr, Adel Bibi |
| date | 2026-06-25 |
| venue | arXiv (preprint, not peer reviewed) |
| id | arXiv:2606.14397 |
| affiliation | University of Oxford |
| upvotes | 20 (Hugging Face Daily Papers, 2026-11-01) |
| links | abs pdf hf code |
| abstract | As agentic systems continue to evolve and are widely deployed in real-world scenarios, there is a growing demand to faithfully evaluate their capabilities. However, current benchmarks are typically built on popular applications with relatively simple tasks and focus on a narrow set of capabilities while overlooking broader dimensions, resulting in saturated performance on modern agents and failing to probe their limitations. To this end, we introduce GauntletBench, a web-based benchmark for evaluating agent generalisation in challenging scenarios, focusing on three underexplored capabilities (temporal perception, graphical understanding, and 3D reasoning), across five less-covered professional applications (Video Editor, Workflow Builder, 3D Modeller, Flight Analyser, and Circuit Designer), each with 20 vision-intensive tasks (100 in total). Our benchmark provides a modular pipeline that comprises an environment compatible with both open- and closed-source agent frameworks, a controlled web-based application, a well-structured task suite, and an automated evaluation engine with diverse metrics. Contrary to widespread expectations, our empirical results reveal that frontier agentic systems remain far from achieving human-level performance. Even the state-of-the-art agent achieves only a 19.1% success rate on our GauntletBench, highlighting the limitations in these overlooked capabilities and generalisation. By comparison, non-expert human annotators achieve over 80% success on our challenging yet feasible tasks, revealing the substantial gap between current agent capabilities and those required for complex real-world scenarios. |
cite
@misc{vysotskyi2026running,
title={Running the Gauntlet: Re-evaluating the Capabilities of Agents Beyond Familiar Environments},
author={Mykola Vysotskyi and Runqi Lin and Grzegorz Biziel and Michal Zakrzewski and Sebastian Montagna and Damian Rynczak and Shreyansh Padarha and Kumail Alhamoud and Zihao Fu and William Lugoloobi and Kai Rawal and Hanna Yershova and Xander Davies and Taras Rumezhak and Guohao Li and Fazl Barez and Baoyuan Wu and Arkadiusz Drohomirecki and Yarin Gal and Chris Russell and Christopher Summerfield and Adam Mahdi and Volodymyr Karpiv and Philip Torr and Adel Bibi},
year={2026},
eprint={2606.14397},
archivePrefix={arXiv},
url={https://arxiv.org/abs/2606.14397}
}