LiveMACEBench scores LLM agents by process in live markets
Original title: LiveMACE: Process-Aware Evaluation of LLM Agent Capabilities in Evolving Markets
Over 30 days, five frontier models' returns often diverged from mechanism-specific capability measurements.
LiveMACEBench evaluates persistent LLM agents in live financial markets using both realized outcomes and diagnostics from complete decision traces. Five frontier models run under matched tool use, persistent memory, rule following, and multi-agent collaboration settings. The authors report an outcome-capability gap: similar returns can come from different patterns of mechanism use.
Unconfirmed: The abstract does not name the five models or report independent replication.
- Evaluation used 30 days of live market trajectories.
- Trace diagnostics separate mechanism access, use, and downstream performance.
- The paper says those three measures are not interchangeable.
- The five frontier models are not named in the abstract.
From the original article
1 min read at huggingface.co
Evaluating agents by outcomes alone can obscure the capabilities that produce them. This problem is especially pronounced in evolving environments, where outcomes reflect a closed-loop interaction between agent behavior and changing external conditions. We introduce LiveMACEBench, a process-aware benchmark that uses live financial markets as a naturally evolving testbed for persistent LLM agents.
By Jun Zhao, Leiming Fu, Yanbo Wen · huggingface.co
A short excerpt; the article and its images belong to the publisher. Summary, key facts and assessment above are AI-written from the article and can be wrong.