Revisiting Certified Defense with Differential Privacy on Vision Transformers
Laplace PixelDP noise after Vision Transformer patch embeddings collapses accuracy; spectral-norm repair voids the usual certificate.
The paper revisits Pixel Differential Privacy certified defenses on Vision Transformers, previously studied mainly on CNNs. Laplace noise injected after patch embedding, using the inherited grouped l1 sensitivity bound, collapses accuracy to chance across noise scales, while the Gaussian mechanism stays trainable. The failure is traced to sensitivity geometry: attenuation from the inherited projection grows with layer width and kernel size. Replacing the l1 constraint with a spectral-norm constraint removes the collapse but breaks the standard Laplace certificate, so the authors derive a dimension-free (epsilon, delta) guarantee for Laplace noise under l2 sensitivity.
- Laplace noise after patch embedding yields chance-level Vision Transformer accuracy.
- The Gaussian mechanism remains trainable under the same injection point.
- Collapse comes from l1 sensitivity geometry, which worsens with width.
- A spectral-norm constraint removes collapse but breaks the Laplace certificate.
- Authors derive a dimension-free epsilon-delta guarantee under l2 sensitivity.
From the original article
1 min read at arxiv.orgCertified defenses that incorporate differential privacy have proven effective on Convolutional Neural Networks (CNNs), furnishing rigorous robustness guarantees against norm-bounded adversaries. However, the certified robustness behavior of Pixel Differential Privacy (PixelDP) remains largely unexplored with the self-attention architecture now dominating the deep-learning landscape.
By Jun Yan · arxiv.org
A short excerpt; the article and its images belong to the publisher. Summary, key facts and assessment above are AI-written from the article and can be wrong.