APort Vault: Benchmarking AI Agent Payment Authorization with the Open Agent Passport
APort Vault benchmark evaluates AI agent payment security across 14 models, showing Open Agent Passport blocks all unauthorized transfers.
APort Vault is a benchmark for evaluating payment authorization in tool-using AI agents, testing 4,371 human-written attacks across 14 models from 8 labs. The Open Agent Passport (OAP) specification successfully blocked all unauthorized transfers, reducing unauthorized payments from 140 to 0. The benchmark includes 225,964 evaluations with public release of data, scoring code, and analysis scripts.
- APort Vault benchmark evaluates payment authorization in AI agents with 4,371 human-written attacks.
- Tested 14 models from 8 labs across 5 policy configurations.
- Open Agent Passport (OAP) specification reduced unauthorized transfers from 140 to 0.
- 62.6% of Level 4 prompts elicited payment requests from all 14 models.
- Products
- APort VaultOpen Agent Passport
- Organizations
- APort
- AI models
- APort Vault
From the original article
1 min read at huggingface.co
APort Vault is a benchmark for payment authorization in tool-using AI agents. It replays 4,371 attacks written by humans against a live payment agent during a public capture-the-flag event, across 14 models from 8 labs, five policy configurations and two replay tracks, with and without a deterministic pre-action check implementing the Open Agent Passport (OAP) specification. 225,964 evaluations completed.
By Uchi Uchibeke · huggingface.co
A short excerpt; the article and its images belong to the publisher. Summary, key facts and assessment above are AI-written from the article and can be wrong.