Skip to content
NewA calmer, faster ZeroHour

PrivacyPR benchmarks privacy-aware code review on real GitHub fixes

Original title: PrivacyPR: A Dataset of Real-World Privacy Fixes for Benchmarking Privacy-Aware Code Review

arXiv cs.CRby Hua Yang ()1 min read1
AI summary · x-ai/grok-4.7Summaries by Vani AI

On a 200-PR subset, coding agents found 42-51% of reintroduced privacy defects unless the prompt asked about privacy.

PrivacyPR is a dataset of 4,131 merged GitHub pull requests from 1,469 repositories, created between June 2018 and January 2024, with commits, diffs, review discussion, and LLM labels for privacy relevance, fix type, and GDPR principles. It includes 1,169 high-confidence privacy fixes, 1,643 non-fix PRs, a manually checked Verified split, and a Small evaluation subset. An evaluation harness reverts each fix to reintroduce the defect. On Small, two coding agents detected 42-51% of defects under a generic review prompt and 88-91% when asked to check privacy, while flagging 11-15% of non-fix PRs.

Unconfirmed: GDPR-principle labels are LLM-generated; human agreement is reported only as 72% endorsement on Verified.

Key facts
  • Verified contains 250 privacy fixes and 250 non-fix PRs; Small is 100 plus 100.
  • Both annotators endorsed 72% of the LLM's GDPR-principle assignments on Verified.
  • Privacy-specific prompts raised detection to 88-91% but still flagged 11-15% of non-fix PRs.
  • Data, code, reviews, and the harness are published at Zenodo doi 10.5281/zenodo.23744765.
Analyze this story with Vani AIImpact, who is affected and what defenders should do, in a new chat

From the original article

1 min read at arxiv.org

Coding agents increasingly review code, yet whether they catch the privacy defects that laws such as the GDPR require software to avoid remains largely untested, partly because few datasets connect real privacy defects to their accepted fixes and review context. We present PrivacyPR, a dataset of 4,131 merged GitHub pull requests (PRs) from 1,469 repositories, created between June 2018 and January 2024.

arXiv cs.CR

By Hua Yang · arxiv.org

Read the full article at arXiv cs.CR

A short excerpt; the article and its images belong to the publisher. Summary, key facts and assessment above are AI-written from the article and can be wrong.