| title | Beyond Success Rate: Cost-Aware Evaluation of Offensive and Defensive Security Agents |
|---|---|
| authors | Paul Kassianik, Blaine Nelson, Yaron Singer |
| date | 2026-07-17 |
| venue | arXiv (preprint, not peer reviewed) |
| id | arXiv:2607.15263 |
| upvotes | 6 (Hugging Face Daily Papers, 2026-11-01) |
| links | abs pdf hf |
| abstract | Security-agent evaluations commonly measure peak offensive capability under generous inference budgets, emphasizing vulnerability discovery, exploit development, penetration testing, and CTF completion. Such measurements are useful but incomplete: in operational security, every reasoning step, tool call, telemetry query, and enrichment request consumes budget. We evaluate language-model security agents through this cost-success lens on offensive Cybench challenges and defensive Splunk BOTS v1 investigation challenges. Instead of reporting only best-case success, we compare models at fixed cost levels and decompose performance by inference spend and tool spend. Our results show distinct scalingregimes for red- and blue-team tasks. Offensive CTF performance improves with additional test-time compute, and scaled open-weight models can approach frontier proprietary systems while remaining cost-competitive. Defensive SOC investigation does not scale in the same way: success depends more heavily on disciplined tool use, telemetry navigation, and selective enrichment than on raw reasoning budget alone. We argue that security-agent benchmarks should measure economic efficiency and operational fit alongside task success. Cost-aware, SOC-native evaluations provide a clearer picture of which models are practically useful today and where defensive agents still need to improve. We present an interactive website with our results https://evals.frontier.security. |
cite
@misc{kassianik2026beyond,
title={Beyond Success Rate: Cost-Aware Evaluation of Offensive and Defensive Security Agents},
author={Paul Kassianik and Blaine Nelson and Yaron Singer},
year={2026},
eprint={2607.15263},
archivePrefix={arXiv},
url={https://arxiv.org/abs/2607.15263}
}