PrivacyPR benchmarks privacy-aware code review on real GitHub fixes
Original title: PrivacyPR: A Dataset of Real-World Privacy Fixes for Benchmarking Privacy-Aware Code Review
On a 200-PR subset, coding agents found 42-51% of reintroduced privacy defects unless the prompt asked about privacy.
PrivacyPR is a dataset of 4,131 merged GitHub pull requests from 1,469 repositories, created between June 2018 and January 2024, with commits, diffs, review discussion, and LLM labels for privacy relevance, fix type, and GDPR principles. It includes 1,169 high-confidence privacy fixes, 1,643 non-fix PRs, a manually checked Verified split, and a Small evaluation subset. An evaluation harness reverts each fix to reintroduce the defect. On Small, two coding agents detected 42-51% of defects under a generic review prompt and 88-91% when asked to check privacy, while flagging 11-15% of non-fix PRs.
Unconfirmed: GDPR-principle labels are LLM-generated; human agreement is reported only as 72% endorsement on Verified.
- Verified contains 250 privacy fixes and 250 non-fix PRs; Small is 100 plus 100.
- Both annotators endorsed 72% of the LLM's GDPR-principle assignments on Verified.
- Privacy-specific prompts raised detection to 88-91% but still flagged 11-15% of non-fix PRs.
- Data, code, reviews, and the harness are published at Zenodo doi 10.5281/zenodo.23744765.
From the original article
1 min read at arxiv.orgCoding agents increasingly review code, yet whether they catch the privacy defects that laws such as the GDPR require software to avoid remains largely untested, partly because few datasets connect real privacy defects to their accepted fixes and review context. We present PrivacyPR, a dataset of 4,131 merged GitHub pull requests (PRs) from 1,469 repositories, created between June 2018 and January 2024.
By Hua Yang · arxiv.org
A short excerpt; the article and its images belong to the publisher. Summary, key facts and assessment above are AI-written from the article and can be wrong.