Region-Level Black-Box Defense Against Stealthy Embedding-Space Backdoors in CLIP
CLIPGuard localizes and inpaints CLIP embedding backdoors, cutting attack success to about 1% in black-box tests.
CLIPGuard is a lightweight black-box defense against embedding-space backdoors in CLIP that does not need model weights, gradients, logits, or clean validation data. It measures segment-wise embedding perturbations and semantically inpaints only suspicious regions. On STL-10, ImageNet, and triggers including BadCLIP, BadNets, blended, patch, and typographic attacks, attack success fell to as low as 1.05% while clean accuracy reached 86.34%, beating CleanCLIP and CleanerCLIP.
- Targets stealthy embedding-space backdoors rather than logit triggers.
- Black-box: no weights, gradients, logits, or clean validation data.
- Inpaints only image segments with suspicious embedding shifts.
- Attack success as low as 1.05% with clean accuracy up to 86.34%.
Full article200 words · extracted from arxiv.org · click to collapse
Contrastive Language--Image Pretraining (CLIP) has emerged as a dominant vision backbone due to its strong transferability and zero-shot capabilities. However, recent studies reveal a critical vulnerability: embedding-space backdoor attacks. By poisoning only a tiny fraction of image--text pairs, adversaries can implant stealthy triggers that induce targeted shifts in CLIP's joint embedding space. Unlike conventional backdoors that manipulate classifier logits, these attacks corrupt representations directly, making them highly effective under extremely low poisoning ratios and difficult to detect. Existing defenses require access to model parameters, gradients, logits, or clean validation data---assumptions that rarely hold in realistic black-box deployments. Moreover, current black-box methods struggle to accurately localize small or out-of-distribution triggers. We propose CLIPGuard, a lightweight and fully black-box defense specifically designed to mitigate embedding-space backdoors in CLIP encoders. CLIPGuard identifies malicious regions by measuring segment-wise embedding perturbations and selectively purifies only suspicious segments via semantic inpainting, preserving benign visual content and alignment quality. Extensive experiments on STL-10, ImageNet, and diverse trigger families---including BadCLIP, BadNets, blended, patch-based, and typographic attacks---demonstrate that CLIPGuard reduces attack success rates to as low as 1.05% while maintaining clean accuracy up to 86.34%, consistently outperforming existing black-box defenses, including CleanCLIP and CleanerCLIP. Our code is available https://github.com/wsu-cyber-security-lab-ai/CLIPGuard.git
Text extracted automatically; images, tables and formatting may be missing. Original: https://arxiv.org/abs/2609.31558