OR-License tests LLM agents on real operations-research client work
Original title: OR-License: Evaluating LLM Agents as Real-World Operations Research Practitioners
Across 31 paid engagements, the strongest Claude Code setup hit 88.39% strict feasibility and a DAQ of 0.87.
OR-License is an end-to-end benchmark of 31 instances from 10 industrial scenarios drawn from paid operations-research engagements totaling USD 3.68 million. Agents must work from de-identified original materials, with ORSim disclosing business facts but not OR expertise. Several of 11 frontier LLMs beat expert references on conditional objective quality, but none matched expert Deployment-Adjusted Quality.
Unconfirmed: Engagement value and expert-relative scores are reported by the paper, not independently verified.
- Hidden checks score feasibility and objective quality against client-accepted expert solutions.
- Task-equalized research-chain completion rates ranged from 24% to 45%.
- Trajectory analysis found formulation and implementation failures.
- Later information-acquisition stages showed higher confirmed issue-resolution proportions.
From the original article
1 min read at huggingface.co
Can large language model (LLM) agents deliver expert-level operations research (OR) solutions directly from original client materials, without prior expert reformulation? Existing OR benchmarks typically begin with self-contained descriptions or predefined optimization tasks, underrepresenting the translation of original business materials into professionally validated solutions.
By Peisong Wang, Feixue Liu, Yize Meng · huggingface.co
A short excerpt; the article and its images belong to the publisher. Summary, key facts and assessment above are AI-written from the article and can be wrong.