| title | PrefixGuard: From LLM-Agent Traces to Online Failure-Warning Monitors |
|---|---|
| authors | Xinmiao Huang, Jinwei Hu, Rajarshi Roy, Changshun Wu, Yi Dong, Xiaowei Huang |
| date | 2026-05-07 |
| venue | arXiv (preprint, not peer reviewed) |
| id | arXiv:2605.06455 |
| affiliation | Trustworthy Autonomous Cyber Physical Systems (ACPS) Lab |
| upvotes | 1 (Hugging Face Daily Papers, 2026-11-01) |
| links | abs pdf hf code |
| abstract | Large language model (LLM) agents now execute long, tool-using tasks where final outcome checks can arrive too late for intervention. Online warning requires lightweight prefix monitors over heterogeneous traces, but hand-authored event schemas are brittle and deployment-time LLM judging is costly. We introduce PrefixGuard, a trace-to-monitor framework with an offline StepView induction step followed by supervised monitor training. StepView induces deterministic typed-step adapters from raw trace samples, and the monitor learns an event abstraction and prefix-risk scorer from terminal outcomes. Across WebArena, τ^2-Bench, SkillsBench, and TerminalBench, the strongest PrefixGuard monitors reach 0.900/0.710/0.533/0.557 AUPRC. Using the strongest backend within each representation, they improve over raw-text controls by an average of +0.137 AUPRC. LLM judges remain substantially weaker under the same prefix-warning protocol. We also derive an observability ceiling on score-based area under the precision-recall curve (AUPRC) that separates monitor error from failures lacking evidence in the observed prefix. For finite-state audit, post-hoc deterministic finite automaton (DFA) extraction remains compact on WebArena and τ^2-Bench (29 and 20 states) but expands to 151 and 187 states on SkillsBench and TerminalBench. Finally, first-alert diagnostics show that strong ranking does not imply deployment utility: WebArena ranks well yet fails to support low-false-alarm alerts, whereas τ^2-Bench and TerminalBench retain more actionable early alerts. Together, these results position PrefixGuard as a practical monitor-synthesis recipe with explicit diagnostics for when prefix warnings translate into actionable interventions. |
cite
@misc{huang2026prefixguard,
title={PrefixGuard: From LLM-Agent Traces to Online Failure-Warning Monitors},
author={Xinmiao Huang and Jinwei Hu and Rajarshi Roy and Changshun Wu and Yi Dong and Xiaowei Huang},
year={2026},
eprint={2605.06455},
archivePrefix={arXiv},
url={https://arxiv.org/abs/2605.06455}
}