Only Pay What You Must Spend: On-Demand Privacy Budget Payment for Differentially Private RAG
Researchers propose SparsePay-RAG, which spends differential-privacy budget only on private increments during retrieval-augmented generation.
The paper argues that existing differentially private RAG systems waste privacy budget on retrieval and generation steps that could use only public information. SparsePay-RAG treats public data as a zero-privacy prior and spends budget only on the private increment. It narrows retrieval with public topic-guided clustering, limits private-access frequency via isotonic cross-layer trajectory fitting, and compresses each access with DP contrastive decoding. Under strong privacy constraints, the authors report better privacy-utility trade-offs than prior DP-RAG baselines.
- SparsePay-RAG charges privacy budget only for the private increment over public data.
- It uses public topic clustering, isotonic trajectory fitting, and DP contrastive decoding.
- Existing DP-RAG methods waste budget on steps that need no private access.
- Under strong privacy constraints, it reports better privacy-utility trade-offs than baselines.
Full article182 words · extracted from arxiv.org · click to collapse
Deploying large language models (LLMs) on sensitive data via Retrieval-Augmented Generation (RAG) introduces severe privacy risks. Recent studies apply Differential Privacy (DP) to LLMs with RAG for formal privacy guarantees. However, existing DP-RAG frameworks rapidly exhaust the privacy budget. Although recent efforts attempt to save the budget by narrowing the retrieval scope or sparsifying private generation, these methods themselves cumulatively consume the budget, whereas they could actually rely merely on public information or at a negligible one-time privacy cost. This mismatch fails to align budget expenditure with the model's actual reliance on private data, causing substantial waste on operations that require no private access. To address this, we propose SparsePay-RAG, adopting "only pay what you must spend" as its core principle. Using public information as a zero-privacy prior, it charges the privacy budget only for the private increment. Specifically, SparsePay-RAG narrows the retrieval scope via public topic-guided clustering, adaptively controls private access frequency without privacy cost through isotonic cross-layer trajectory fitting, and compresses per-access budget via DP contrastive decoding. Under strong privacy constraints, experiments show SparsePay-RAG achieves superior privacy-utility trade-offs over baselines.
Text extracted automatically; images, tables and formatting may be missing. Original: https://arxiv.org/abs/2609.27406