GARD cuts CLIP visual backdoor success below 1%
Original title: GARD: Gradient-Guided Adversarial Representation Defense against Backdoors in Multimodal Contrastive Learning Models
The post-hoc defense covers six visual backdoor attacks on two CLIP backbones and needs no extra inference compute.
Researchers propose GARD, a target- and trigger-agnostic repair for backdoored multimodal contrastive encoders such as CLIP. It uses complementary global and local adversarial probes to displace clean-image representations, then realigns them with detached clean anchors while keeping image-text alignment. Across 12 attack-backbone combinations, attack success falls below 1% with competitive clean accuracy.
- Existing clean fine-tuning can leave backdoors active.
- Trigger inversion depends on estimating the target and trigger.
- GARD is evaluated on six visual backdoor attacks and two CLIP backbones.
- Attack success is below 1% in all 12 combinations.
From the original article
1 min read at arxiv.orgMultimodal contrastive learning models such as CLIP are vulnerable to backdoor attacks with diverse trigger structures, ranging from localized patches to globally distributed perturbations, with recent attacks increasingly optimizing these triggers in the representation space.
By Yuhong Wang · arxiv.org
A short excerpt; the article and its images belong to the publisher. Summary, key facts and assessment above are AI-written from the article and can be wrong.