Beyond Success Rate: Cost-Aware Evaluation of Offensive and Defensive Security Agents

2026-07-16 · 17:54 UTC ·Paul Kassianik, Blaine Nelson, Yaron Singer ·1 min read

Brief

Cost-aware evaluation of LLM-based security agents measures performance at fixed monetary/compute budgets and decomposes costs into inference and tool spend. The authors evaluate offensive Cybench CTFs and defensive Splunk BOTS v1 SOC investigations, finding offensive tasks scale with test-time compute—scaled open-weight models approach proprietary frontiers cost-competitively—whereas defensive success hinges on disciplined tool use, telemetry navigation, and selective enrichment rather than raw reasoning budget.

Why it matters

Authors Paul Kassianik, Blaine Nelson, and Yaron Singer (arXiv 2026-07-16) propose a cost-aware evaluation that compares security agents at fixed cost levels and decomposes performance into inference spend versus tool (telemetry/enrichment) spend; results and an interactive site are at https://evals.frontier.security.

Key details

On offensive Cybench CTF challenges, performance improves with additional test-time compute; the paper reports that scaled open-weight models can approach frontier proprietary systems while remaining cost-competitive.

On defensive Splunk BOTS v1 SOC investigations, success depends more on disciplined tool use, telemetry navigation, and selective enrichment than on raw reasoning/inference budget, so economic efficiency and operational fit matter more than peak success rate.

Abstract

Security-agent evaluations commonly measure peak offensive capability under generous inference budgets, emphasizing vulnerability discovery, exploit development, penetration testing, and CTF completion. Such measurements are useful but incomplete: in operational security, every reasoning step, tool call, telemetry query, and enrichment request consumes budget. We evaluate language-model security agents through this cost-success lens on offensive Cybench challenges and defensive Splunk BOTS v1 investigation challenges. Instead of reporting only best-case success, we compare models at fixed cost levels and decompose performance by inference spend and tool spend. Our results show distinct scalingregimes for red- and blue-team tasks. Offensive CTF performance improves with additional test-time compute, and scaled open-weight models can approach frontier proprietary systems while remaining cost-competitive. Defensive SOC investigation does not scale in the same way: success depends more heavily on disciplined tool use, telemetry navigation, and selective enrichment than on raw reasoning budget alone. We argue that security-agent benchmarks should measure economic efficiency and operational fit alongside task success. Cost-aware, SOC-native evaluations provide a clearer picture of which models are practically useful today and where defensive agents still need to improve. We present an interactive website with our results https://evals.frontier.security.