FrontierCS 2 benchmarks LLM agents on 48 open-ended CS tasks
Original title: FrontierCS 2: A Benchmark for Computer Science R&D
Claude Opus 5.5 leads 12 agents with a 58% pass rate, and 16 tasks have no passing run.
FrontierCS 2 puts agents in a research repository and lets them submit patches, unlike benchmarks that grade one program from a problem statement. A private judge returns a development-suite score and grades the final submission against a hidden human reference. The 48 tasks span 13 domains. Beating the reference on the combined score is not a pass if some metrics or workloads still fail.
- The evaluator, held-out final suite and reference stay private.
- A private set of more than 100 tasks is not yet released.
- Most runs that beat the reference still miss some metric or workload.
- The median task is about as hard as the strongest agent can handle.
From the original article
1 min read at huggingface.co
Can LLM agents make contributions to frontier computer science R&D? We introduce FrontierCS 2, a benchmark of 48 open-ended tasks. We take a paper and the code its authors released and delete the paper's contribution.
By Qiuyang Mang, Wenhao Chai, Runyuan He · huggingface.co
A short excerpt; the article and its images belong to the publisher. Summary, key facts and assessment above are AI-written from the article and can be wrong.