Region-Level Black-Box Defense Against Stealthy Embedding-Space Backdoors in CLIP
CLIPGuard localizes and inpaints CLIP embedding backdoors, cutting attack success to about 1% in black-box tests.
CLIPGuard is a lightweight black-box defense against embedding-space backdoors in CLIP that does not need model weights, gradients, logits, or clean validation data. It measures segment-wise embedding perturbations and semantically inpaints only suspicious regions. On STL-10, ImageNet, and triggers including BadCLIP, BadNets, blended, patch, and typographic attacks, attack success fell to as low as 1.05% while clean accuracy reached 86.34%, beating CleanCLIP and CleanerCLIP.
- Targets stealthy embedding-space backdoors rather than logit triggers.
- Black-box: no weights, gradients, logits, or clean validation data.
- Inpaints only image segments with suspicious embedding shifts.
- Attack success as low as 1.05% with clean accuracy up to 86.34%.
From the original article
1 min read at arxiv.orgContrastive Language--Image Pretraining (CLIP) has emerged as a dominant vision backbone due to its strong transferability and zero-shot capabilities. However, recent studies reveal a critical vulnerability: embedding-space backdoor attacks. By poisoning only a tiny fraction of image--text pairs, adversaries can implant stealthy triggers that induce targeted shifts in CLIP's joint embedding space.
By Ahmed Abdelnaby · arxiv.org
A short excerpt; the article and its images belong to the publisher. Summary, key facts and assessment above are AI-written from the article and can be wrong.