EvoRiskBench: An Evolving Benchmark for Runtime Security Risks in Workspace Agents
EvoRiskBench releases 450 adversarial workspace-agent tasks; worst configuration (Codex plus DeepSeek-V4-Pro-0813) reached 68.44% attack success rate.
EvoRiskBench is an evolving benchmark for runtime security risks in workspace agents, organized around an EP-Path-EF framework linking nine entry-point categories to five effect categories through agent-mediated risk paths, validated by a 20-participant interpretability study. An automated workflow constructs and executes 450 adversarial tasks across six scenarios in isolated environments, verifying outcomes via runtime traces and environment states. Across nine model-harness configurations (GPT-5.6 Sol, DeepSeek-V4-Pro-0813, Claude Opus 5 with Claude Code, Codex, and OpenClaw harnesses), the most vulnerable, Codex with DeepSeek-V4-Pro-0813, reached a 68.44% attack success rate. ASR varied more across models than harnesses, and harness differences depended on the model.
- 450 adversarial tasks across six scenarios, verified via runtime traces and environment states
- Worst configuration: Codex with DeepSeek-V4-Pro-0813 at 68.44% attack success rate
- ASR varies more across models than harnesses; harness effect depends on model
- EP-Path-EF framework defines nine entry-point and five effect categories
- Dataset and evaluation platform to be released after safety checks
Full article220 words · extracted from arxiv.org · click to collapse
Workspace agents combine large language models with execution harnesses to perform stateful, multi-step tasks that access or modify external resources. Existing benchmarks leave gaps in executable coverage of their runtime security risks, while evolving model capabilities, harnesses, tools, and threats motivate benchmark evolution. We introduce EvoRiskBench, an evolving benchmark organized around the EP-Path-EF framework, which links an initial risk entry point to a one-hop technical effect through an agent-mediated risk path. The framework defines nine entry-point categories and five effect categories; a 20-participant study supports their interpretability and classification consistency on representative cases. Guided by this framework, an automated end-to-end workflow constructs and executes risk cases in isolated environments and independently verifies outcomes using runtime traces and environment states. The benchmark provides a reproducible dataset of 450 adversarial tasks across six scenarios. We evaluate nine model-harness configurations spanning three models (GPT-5.6 Sol, DeepSeek-V4-Pro-0813, and Claude Opus 5) and three harnesses (Claude Code, Codex, and OpenClaw). Our results reveal substantial vulnerabilities across systems. The most vulnerable configuration, Codex with DeepSeek-V4-Pro-0813, reaches a 68.44% attack success rate (ASR), indicating that configuration of workspace agent is insufficient to ensure secure autonomous execution. ASR varies more across models than harnesses, and harness differences depend on the model. The benchmark cases and evaluation platform will be released after completion of artifact safety and reproducibility checks.
Text extracted automatically; images, tables and formatting may be missing. Original: https://arxiv.org/abs/2610.03153