Cross-modal triggers activate backdoors in multimodal LLMs
Original title: Under the Skin: Backdoor Infection in Multimodal Large Language Models
From a backdoor in one modality, the method builds a trigger in another without poisoning that modality during training.
Researchers introduce backdoor infection, a cross-modal attack on multimodal large language models. A model already backdoored in one modality can be activated by a trigger in another modality through the shared latent space, without modifying that second modality in training. A lightweight constraint-aware optimizer generates the triggers. Experiments on LLaVA, InternVL, Qwen-VL, and Gemma found them more efficient and less detectable than direct optimization baselines, according to the authors.
Unconfirmed: Effectiveness and detectability claims come only from the authors' experiments.
- The infected modality is not modified or poisoned during training.
- A constraint-aware optimizer is used to generate practical triggers.
- Tested models include LLaVA, InternVL, Qwen-VL, and Gemma.
From the original article
1 min read at arxiv.orgMultimodal Large Language Models (MLLMs) have achieved impressive performance in generating text conditioned on inputs from different modalities. In this work, we introduce backdoor infection, a cross-modal attack framework targeting MLLMs. Starting from a model already injected with a backdoor in one modality, our attack constructs a corresponding trigger in another modality, called the infected modality.
By Jiawen Wang · arxiv.org
A short excerpt; the article and its images belong to the publisher. Summary, key facts and assessment above are AI-written from the article and can be wrong.