DolphinBench: Mapping the Pareto Frontier of Agent Memory
DolphinBench scores agent memory by task success, cost, and latency across three personas.
DolphinBench measures agent memory through task completion rather than prompted question answering. It uses three knowledge-work personas, each with roughly 500,000 tokens of user messages, and 200 verified tasks per persona that succeed only when the relevant history is available. Every evaluation must report total cost and latency with accuracy so memory systems can be compared along the accuracy-cost frontier.
- Three personas each have about 500,000 tokens of user-message history.
- Two hundred tasks per persona must succeed with history and fail without it.
- Submissions must report total cost and latency alongside accuracy.
- No prior memory benchmark combines task completion, cost, and latency.
Full article167 words · extracted from arxiv.org · click to collapse
Agents today often take real-world actions that depend on long-term memory and context recall over time. However, most current memory benchmarks are built for a conversational question-answer format, where the question itself signals that some fact must be retrieved, and often which one. Moreover, benchmarks rarely require anything beyond accuracy from submissions, allowing memory systems to make unreasonable cost/time tradeoffs to achieve higher scores. We present DolphinBench, a benchmark that evaluates memory directly through an agent's task completion. DolphinBench includes three knowledge-work personas with roughly 500k tokens of user messages per persona and evaluates agents on tasks that depend on information from that history. We verify all 200 tasks per persona by running an agent with and without the relevant history, requiring success with it and failure without it. Finally, we require all evaluations to report total cost and latency alongside accuracy, which enables us to evaluate agent memory systems holistically. No existing memory benchmark combines all three. The dataset and evaluation code are available at https://dolphinbench.ai.
Text extracted automatically; images, tables and formatting may be missing. Original: https://arxiv.org/abs/2609.24971