FINCOSMOS evaluates finance agents without fixed tasks, data, or timelines
Original title: FINCOSMOS: An Open-World Evaluation Paradigm for Agents in Finance
In 150 sessions no run passed; agents surfaced only 54-64% of untold details, and experts blamed them for 88.33% of failures.
FINCOSMOS is an open-world evaluation for language agents doing financial work. A simulated user holds a hidden strategy and reveals it only when asked. The agent must acquire web data, backtest under a data cutoff, and deliver results that programs can verify without a reference answer. Across four coding agents and a tool-free model in 150 sessions, no session was accepted. With the benchmark's feedback, an agent with persistent memory learned all ten tasks.
- A run is accepted only if the user is satisfied and no veto check fails.
- Every session the simulated user signed off still failed a program check.
- Agents raised only 54-64% of details they were not told.
- Human experts attributed 88.33% of failures to the agents.
- Persistent memory plus FINCOSMOS feedback let an agent learn all ten tasks.
From the original article
1 min read at huggingface.co
Language agents increasingly work for users on the open Web, and finance is where trusting them is hardest. The data must come from the Web, the strategy lives in the user's head, and correctness depends on time, so an agent can fail silently: a strategy the user never asked for, unverified data, or a backtest that uses the future yields a report as convincing as a correct one.
By Yupeng Cao, Xueqing Peng, Lingfei Qian · huggingface.co
A short excerpt; the article and its images belong to the publisher. Summary, key facts and assessment above are AI-written from the article and can be wrong.