FinAutoRubric: Expert-Guided Automatic Rubric Generation for Evaluating Financial Research Agents
FinAutoRubric generates expert-guided, cutoff-aware rubrics for evaluating financial research agents.
FinAutoRubric lets experts specify reusable evaluation guidance while agents and code generate, review, and validate query-specific rubrics that respect an information cutoff. A writer researches expected values and a reviewer checks them, escalating failures to humans, with a Task Bank carrying reusable criteria across tasks. On three expert finance benchmarks, its rubrics match top generators, agree with human grading, and were preferred by in-house analysts in a blind review. The released 100-query benchmark spans 78 tasks and eight asset classes.
- Experts set reusable guidance while agents generate query-specific rubrics.
- Writer and reviewer agents check values and escalate failures to humans.
- Rubrics track expert scores and state more expected criterion values.
- Released benchmark covers 100 queries, 78 tasks, and eight asset classes.
From the original article
1 min read at arxiv.orgEvaluating finance research agents requires rubrics that reflect expert standards and fix the values correct as of an information cutoff. Expert-reviewed finance benchmarks rely on fixed, per-item rubrics, which are costly to extend and cannot encode each institution's own standard.
By Hoyoung Lee · arxiv.org
A short excerpt; the article and its images belong to the publisher. Summary, key facts and assessment above are AI-written from the article and can be wrong.