PIA-Bench: Towards Automated Privacy Impact Assessment with Large Language Models
Researchers release PIA-Bench, the first open benchmark evaluating how accurately LLMs can automate privacy impact assessments using 73 curated federal PIAs.
PIA-Bench is the first open benchmark for evaluating large language models on real-world privacy impact assessments (PIAs). The authors audited 499 expert-authored PIAs published by US federal agencies and curated 73 structured PIAs comprising 451 privacy risk items and 831 mitigation items. Off-the-shelf LLMs were found to produce meaningful assessments while identifying clear avenues for improvement. The paper calls for domain-specific LLM agent workflows, accountable LLM infrastructure, and new quality standards for PIAs.