MobileCybench: Evaluating Agent Vulnerability Discovery via Executable Probes
MobileCybench uses 495 executable probes to score AI agents finding flaws in 13 Android apps.
MobileCybench evaluates AI vulnerability reports by replaying exploits against 13 Android applications and running 495 author-reviewed executable probes of security properties. Five coding agents were tested as a malicious on-device app or a low-privilege remote attacker, with either an obfuscated APK or source code. OpenCode with GPT-5.6-Sol triggered probes in 53.8% of apps from an obfuscated APK in the malicious-app setting and 16.7% as a remote attacker. Source access raised the overall trigger rate from 28.8% to 32.8%, and the benchmark surfaced 23 previously unreported vulnerabilities, most confirmed by maintainers.
- Probes encode security properties and replay reported exploits against the app.
- Benchmark covers 13 Android apps and 495 author-reviewed probes.
- Top agent triggered probes in 53.8% of apps from an obfuscated APK.
- The work surfaced 23 previously unreported vulnerabilities, most maintainer-confirmed.
- Products
- AndroidOpenCodeClaude Code
From the original article
1 min read at arxiv.orgAI agents now report vulnerabilities faster than maintainers can review them. Reports often depend on security properties specific to the application, and require considerable human labor to process. To mitigate this, we introduce a framework for evaluating vulnerability reports via probes, executable checks of security properties.
By Andy K. Zhang · arxiv.org
A short excerpt; the article and its images belong to the publisher. Summary, key facts and assessment above are AI-written from the article and can be wrong.