Paper bounds private sparse reward estimation from preference feedback
Original title: Differentially Private Sparse Reward Estimation with Preference Feedback
Sample-DP and Label-DP rates depend on sparse features s* rather than full model size d.
A new theoretical paper studies differentially private sparse reward estimation for RLHF under the Bradley-Terry-Luce model. It assumes preferences depend on s* features far smaller than model dimension d, and gives algorithms for Sample-DP and Label-DP. The authors report upper and lower bounds and say the method beats dense baselines on sentiment generation and safety alignment.
- Sample-DP squared error is Õ(s*/n + (s*)²/n²ε²).
- Label-DP error is Õ(s*/n + s*/nε²) using randomized response.
- The Sample-DP rate is nearly optimal, the paper says.
- Label-DP cannot be improved inside the randomized-response framework.
- Tests cover sentiment generation and safety alignment benchmarks.
From the original article
1 min read at arxiv.orgReinforcement Learning with Human Feedback (RLHF) has become a central paradigm for aligning Large Language Models with human values. However, the reliance on sensitive preference data necessitates rigorous differential privacy (DP) guarantees to prevent user information leakage.
By Meng Ding · arxiv.org
A short excerpt; the article and its images belong to the publisher. Summary, key facts and assessment above are AI-written from the article and can be wrong.