JockeyBench finds LLM racing agents lose under capped-Kelly rules
Original title: JockeyBench: Evaluating LLM Agents on Horse-Racing Decision-Making
On 324 historical Japanese races, all 28 model-scaffold setups had negative mean profit, despite some revision gains.
JockeyBench is a benchmark and protocol for repeated event forecasting using historical Japanese horse racing. It uses time-gated evidence, priced tools, frozen-program controls, and two workflows: an LLM-written predictor or a fresh per-race forecast. Across 28 configurations on 324 races, seven revision gains and two declines survived multiple-testing correction, but every setup lost on average under capped-Kelly settlement. In a four-model comparison, the reusable-program workflow had higher mean betting profit and lower research fees, though both workflows still lost.
- Evaluation separates forecast quality, net utility, and retrospective betting returns and risk.
- Paired tests found seven revision gains and two declines after correction.
- Mean probability bias reverses on the bets that were selected.
- Profitable individual runs can depend on concentrated returns.
From the original article
1 min read at huggingface.co
Repeated event forecasting requires extracting predictive signals from noisy evidence and evaluating their value across successive decisions. We introduce JockeyBench, a benchmark and evaluation protocol that studies this problem through historical Japanese horse racing.
By Zhongwei Xie, Xingsu Li, Jingqiao Yu · huggingface.co
A short excerpt; the article and its images belong to the publisher. Summary, key facts and assessment above are AI-written from the article and can be wrong.