APort Vault: Benchmarking AI Agent Payment Authorization with the Open Agent Passport
APort Vault shows a passport check stopped unauthorized payment recipients across 225,964 agent evaluations.
APort Vault replays 4,371 human-written attacks from a public capture-the-flag against a live payment agent. The study covers 14 models from eight labs, five policy levels, and two replay tracks, with and without an Open Agent Passport pre-action check, totaling 225,964 evaluations. At levels 2–4, transfers to recipients the passport did not permit were 140 of 76,842 with the model alone and zero of 69,297 behind the check. The authors release the evaluations, passports, and scoring code.
- Replayed 4,371 human-written CTF attacks against a payment agent
- Fourteen models, five policies, and 225,964 completed evaluations
- Unauthorized transfers: 140 model-alone versus zero behind the passport
- 25,370 allowed payments executed; 187 transfer calls denied
- Evaluations and scoring code released on Hugging Face
- Products
- APort VaultOpen Agent Passport
- Organizations
- APort
From the original article
1 min read at arxiv.orgAPort Vault is a benchmark for payment authorization in tool-using AI agents. It replays 4,371 attacks written by humans against a live payment agent during a public capture-the-flag event, across 14 models from 8 labs, five policy configurations and two replay tracks, with and without a deterministic pre-action check implementing the Open Agent Passport (OAP) specification. 225,964 evaluations completed.
By Uchi Uchibeke · arxiv.org
A short excerpt; the article and its images belong to the publisher. Summary, key facts and assessment above are AI-written from the article and can be wrong.