MobileCybench: Evaluating Agent Vulnerability Discovery via Executable Probes
MobileCybench uses 495 executable probes to score AI agents finding flaws in 13 Android apps.
MobileCybench evaluates AI vulnerability reports by replaying exploits against 13 Android applications and running 495 author-reviewed executable probes of security properties. Five coding agents were tested as a malicious on-device app or a low-privilege remote attacker, with either an obfuscated APK or source code. OpenCode with GPT-5.6-Sol triggered probes in 53.8% of apps from an obfuscated APK in the malicious-app setting and 16.7% as a remote attacker. Source access raised the overall trigger rate from 28.8% to 32.8%, and the benchmark surfaced 23 previously unreported vulnerabilities, most confirmed by maintainers.
- Probes encode security properties and replay reported exploits against the app.
- Benchmark covers 13 Android apps and 495 author-reviewed probes.
- Top agent triggered probes in 53.8% of apps from an obfuscated APK.
- The work surfaced 23 previously unreported vulnerabilities, most maintainer-confirmed.
Full article245 words · extracted from arxiv.org · click to collapse
AI agents now report vulnerabilities faster than maintainers can review them. Reports often depend on security properties specific to the application, and require considerable human labor to process. To mitigate this, we introduce a framework for evaluating vulnerability reports via probes, executable checks of security properties. A reported exploit is evaluated by replaying it against the application and running the probes: a triggered probe indicates both that the exploit succeeded and which security property it violated. As a probe encodes a security property rather than a known vulnerability, it can detect vulnerabilities that were not known when the probe was written. We instantiate the framework as MobileCybench, a benchmark for vulnerability discovery by AI agents in 13 Android applications, with 495 probes written and reviewed by the authors. We evaluate 5 coding agents (OpenCode with GPT-5.5, GPT-5.6-Sol, and GLM-5.2; Claude Code with Opus 4.8 and Opus 5) under 4 settings: as a malicious app on the victim's device or as a remote attacker with a low-privilege account, each with either only an obfuscated APK or access to the application's source code. Given only the obfuscated APK, the top agent, OpenCode with GPT-5.6-Sol, triggers probes in 53.8% of applications in the malicious-app setting and 16.7% in the remote-attacker setting. With source code, the trigger rate across all agents and both attack settings increases from 28.8% to 32.8%. Building and running the benchmark surfaced 23 previously unreported vulnerabilities, the majority of which have been confirmed by maintainers.
Text extracted automatically; images, tables and formatting may be missing. Original: https://arxiv.org/abs/2609.23980