LiveMACE: Process-Aware Evaluation of LLM Agent Capabilities in Evolving Markets
LiveMACEBench evaluates persistent LLM agents in live markets, separating returns from capability-specific mechanism use.
Researchers introduce LiveMACEBench, a process-aware benchmark that runs persistent LLM agents in live financial markets for 30 days. Five frontier models are compared under matched tool use, persistent memory, rule following, and multi-agent collaboration settings. Realized returns often diverge from mechanism-specific diagnostics taken from complete decision traces, so similar outcomes can reflect different capability patterns. The authors argue that mechanism access, effective use, and downstream performance are not interchangeable measures.
- LiveMACEBench tracks persistent agents across 30 days of live markets.
- Five frontier LLMs are tested under four matched capability setups.
- Realized returns often diverge from trace-level capability diagnostics.
- Similar outcomes can come from different mechanism-use patterns.
From the original article
1 min read at arxiv.orgEvaluating agents by outcomes alone can obscure the capabilities that produce them. This problem is especially pronounced in evolving environments, where outcomes reflect a closed-loop interaction between agent behavior and changing external conditions. We introduce LiveMACEBench, a process-aware benchmark that uses live financial markets as a naturally evolving testbed for persistent LLM agents.
By Jun Zhao · arxiv.org
A short excerpt; the article and its images belong to the publisher. Summary, key facts and assessment above are AI-written from the article and can be wrong.