Selective Channel Restoration for Backdoored Vision-Language Models
Post-training defense restores fragile VLM projector channels, driving backdoor success near zero without inference overhead.
Researchers propose Perturb-Select-Restore (PSR), a post-training defense for vision-language models backdoored through poisoned fine-tuning data. PSR exploits projection fragility: backdoored projectors react more strongly to bounded perturbations than clean ones. It restores the most sensitive output channels in each projection layer to their pretrained values, using sparse updates and adding no inference-time computation. Experiments report attack success rates reduced to near zero while clean-task performance is preserved.
- PSR sparsely restores sensitive projector channels to pretrained values.
- Backdoored projectors are more perturbation-sensitive than clean ones.
- Attack success falls near zero with no added inference cost.
- Clean-task performance is largely preserved across tested tasks.
From the original article
1 min read at arxiv.orgVision-language models (VLMs) exhibit strong multimodal capabilities but remain vulnerable to backdoors implanted through poisoned fine-tuning data. Existing defenses often require extensive parameter updates during fine-tuning or incur per-query overhead during inference.
By Shuming Liu · arxiv.org
A short excerpt; the article and its images belong to the publisher. Summary, key facts and assessment above are AI-written from the article and can be wrong.