| title | Where Did It Go Wrong? Process-Level Evaluation of Web Agents with Semantic State Tracking |
|---|---|
| authors | Jiwan Chung, JiHyuk Byun, Vibhav Vineet, Seon Joo Kim |
| date | 2026-04-08 |
| venue | arXiv (preprint, not peer reviewed) |
| id | arXiv:2606.15673 |
| affiliation | Yonsei University |
| upvotes | 12 (Hugging Face Daily Papers, 2026-11-01) |
| links | abs pdf hf |
| abstract | Web agents act through long interaction sequences, yet existing benchmarks evaluate only terminal success, discarding all process information and offering little guidance on improvement. In this work, we conduct a process-level analysis of web agents. We introduce WebStep, a benchmark of 1,800 task instances with controlled difficulty and automatic semantic state tracking. Each website exposes a deterministic semantic MDP alongside the GUI: the agent operates on the interface, while the environment records high-level states and transitions in the background, enabling fine-grained analysis without manual annotation. Based on the semantic trajectory, we first show that process metrics reveal differences invisible to outcome evaluation: three agents whose success rates cluster within 31-33% diverge in exploration reach versus execution accuracy. Then, decomposing by skill characterizes the nature of these differences, exposing opposite per-skill rankings hidden within the same website: e.g., on Housing, OpenAI CUA outperforms Qwen3.5 by 23.7% on commit actions yet underperforms it by 15.6% on filtering, pinpointing a concrete skill to improve even within a domain. Bifurcation analysis further localizes the decisive error that loses the task and shows that this error is agent-specific rather than shared. Finally, these differences widen as tasks grow harder: success rate is similar on easy tasks but separates sharply as exploration becomes more demanding. Our process-level analysis opens a new avenue in web agent evaluation, providing fine-grained and actionable insight into where and how each agent should be improved. |
cite
@misc{chung2026where,
title={Where Did It Go Wrong? Process-Level Evaluation of Web Agents with Semantic State Tracking},
author={Jiwan Chung and JiHyuk Byun and Vibhav Vineet and Seon Joo Kim},
year={2026},
eprint={2606.15673},
archivePrefix={arXiv},
url={https://arxiv.org/abs/2606.15673}
}