GitHub’s ReviewBench puts AI code reviewers to the test
GitHub launched ReviewBench, a 219-pull-request benchmark for testing AI code-review agents.
GitHub released ReviewBench in research preview to compare AI code-review agents on 219 pull requests from 187 public repositories and 19 programming languages. A three-stage golden set merges findings from human reviewers, follow-up commits, analysis tools, and models, then scores them with a shared rubric. Grounded precision, recall, and F1 measure overlap with known issues, while augmented metrics use an AI judge for additional findings. GitHub said a Copilot code-review lite ensemble raised production recall 13.6 percent, increased comments that prompted code changes by 8 percent, and cut cost per review 8 percent versus a single-reviewer control.
- Tests 219 pull requests from 187 repositories and 19 languages.
- Golden set merges human, commit, tool, and model findings.
- Grounded recall is the main metric for comparing systems.
- Copilot lite ensemble raised production recall by 13.6 percent.
- Products
- ReviewBenchGitHub Copilot
From the original article
3 min read at helpnetsecurity.com
GitHub’s ReviewBench measures how well AI code review tools detect problems before software is released. Available in research preview through its website, the benchmark lets users compare review agents and evaluate their own tools. Code review involves checking proposed changes for mistakes.
By Anamarija Pogorelec · helpnetsecurity.com
A short excerpt; the article and its images belong to the publisher. Summary, key facts and assessment above are AI-written from the article and can be wrong.