RubricAtlas mines type-level rubrics to score deep research agents

Built from 3,079 Chinese financial reports across 99 types, judge pipelines agreed at kappa 0.71 and ranking tau 0.87.

RubricAtlas derives type-level analytical requirements from expert reports on different objects of the same research type, then checks deep research agents against those recurring questions rather than only query-specific fact lists. Completeness is scored separately from argument validity, and binary checkpoints are linked to report passages. The rubrics come from 3,079 Chinese financial research reports covering 99 types. Single reports cover about half of a type's analytical questions, and which reports supply the requirements changes scores and agent rankings. Across ten agents, two judge pipelines reached checkpoint agreement of kappa 0.71 and ranking agreement of Kendall tau 0.87. Controlled deletions lowered scores on the affected analyses; padding did not raise them.