APort Vault: Benchmarking AI Agent Payment Authorization with the Open Agent Passport
APort Vault shows a passport check stopped unauthorized payment recipients across 225,964 agent evaluations.
APort Vault replays 4,371 human-written attacks from a public capture-the-flag against a live payment agent. The study covers 14 models from eight labs, five policy levels, and two replay tracks, with and without an Open Agent Passport pre-action check, totaling 225,964 evaluations. At levels 2–4, transfers to recipients the passport did not permit were 140 of 76,842 with the model alone and zero of 69,297 behind the check. The authors release the evaluations, passports, and scoring code.
- Replayed 4,371 human-written CTF attacks against a payment agent
- Fourteen models, five policies, and 225,964 completed evaluations
- Unauthorized transfers: 140 model-alone versus zero behind the passport
- 25,370 allowed payments executed; 187 transfer calls denied
- Evaluations and scoring code released on Hugging Face
Full article281 words · extracted from arxiv.org · click to collapse
APort Vault is a benchmark for payment authorization in tool-using AI agents. It replays 4,371 attacks written by humans against a live payment agent during a public capture-the-flag event, across 14 models from 8 labs, five policy configurations and two replay tracks, with and without a deterministic pre-action check implementing the Open Agent Passport (OAP) specification. 225,964 evaluations completed. We report five distinct events per evaluation, because collapsing them is how an agent benchmark produces a number that does not survive review. Requests are common and their rate differs far more across configurations than across models, though each attack exists at exactly one configuration so policy and attack cohort vary together: 10.9% of model-alone evaluations at Level 1, 3.0% at Level 2, 0.1% at Level 3, 79.4% at Level 4. On the 1,293 Level 4 prompts, each evaluated on every model, request rates run from 71.2% to 84.3%, and 809 prompts (62.6%) elicited a request from all fourteen models, each ending in a successful payment to the level's allowlisted recipient. The authorization boundary is where the conditions diverge. At Levels 2 to 4, transfers to recipients the passport did not permit number 140 of 76,842 with the model alone and 0 of 69,297 behind the layer, and 105 against 0 on 68,970 matched model, prompt and track triples. The zero spans 790 source sessions, giving a per-session upper bound of 0.38%. It was not obtained by refusing payments: 25,370 payments executed behind the layer, while the policy denied 187 of the 25,640 transfer calls it evaluated, 148 of them for a forbidden recipient. We release the 225,964 evaluations, the level passports, the scoring code and the analysis script at huggingface.co/datasets/aporthq/vault-benchmark-v1 .
Text extracted automatically; images, tables and formatting may be missing. Original: https://arxiv.org/abs/2609.22076