Evaluating Coding Agents on Kernel Exploit Generation
KEX-bench finds coding agents rarely turn kernel bugs into exploit primitives without a reference PoC.
KEX-bench evaluates coding agents on generating exploit primitives against real operating-system kernels, with 45 tasks covering 40 Linux and Windows CVEs. Each task runs in an isolated virtual machine and uses a deterministic verifier for leaks, instruction-pointer control, heap read and write, and arbitrary address write. Without a reference proof of concept, the strongest configuration solved 1 of 20 Windows tasks and 14 of 25 Linux tasks; with a reference PoC it solved 31 of 45. The authors report that agents often reach kernel crashes but fail to shape kernel state into exploit primitives.
- Forty-five isolated-VM tasks cover 40 Linux and Windows kernel CVEs.
- Primitives include leaks, instruction-pointer control, heap access, and arbitrary writes.
- Without a reference PoC, best scores were 5% Windows and 56% Linux.
- Supplying a reference PoC raised the best result to 31 of 45 tasks.
Full article165 words · extracted from arxiv.org · click to collapse
Coding agents now find real vulnerabilities in production software. However, bug discovery results do not measure whether agents can construct exploit primitives. We introduce KEX-bench, a benchmark for evaluating coding agents on exploit primitive generation against real operating-system kernels. KEX-bench contains 45 task instances across 40 Linux and Windows CVEs, covering kernel address leak, instruction-pointer control, heap read, heap write, and arbitrary address write. Each task runs in an isolated virtual machine, exposes controlled tools, and uses a deterministic verifier to check primitive-specific success. We evaluate state-of-the-art coding agents paired with frontier and open-weight models under fixed tool-call budgets. Without a reference proof of concept (PoC), the strongest configuration solves 1 of 20 Windows tasks (5.0%) and 14 of 25 Linux tasks (56.0%). With a reference PoC, the strongest configuration solves 31 of 45 tasks (68.9%). This highlights the gap where agents reach kernel crashes but fail to shape kernel state into exploit primitives. We release KEX-bench for reproducible research on AI-assisted exploitation at https://kex-bench.github.io.
Text extracted automatically; images, tables and formatting may be missing. Original: https://arxiv.org/abs/2609.25591