SteerProbe: Learning to Bypass Safety Steering in Vision-Language Models
SteerProbe bypasses vision-language model safety steering, lifting average harmful responses from 7.43% to 18.36%.
The paper shows that activation-steering defenses for vision-language models can be bypassed by intent-preserving textual, visual, and joint reformulations. SteerProbe is an output-only black-box attack that learns effective reformulations using 500 calibration queries per endpoint and benchmark. Across three VLM backbones, two benchmarks, and three steering defenses, average Harmful Rate rises from 7.43% to 18.36%.
- Intent-preserving text, image, and joint rewrites bypass steering.
- SteerProbe is output-only and uses 500 calibration queries.
- Average harmful rate rises from 7.43% to 18.36%.
- Tests cover three VLMs, two benchmarks, and three defenses.
From the original article
1 min read at arxiv.orgActivation steering offers an inference-time defense for vision--language models (VLMs) by modifying intermediate representations without updating backbone parameters. However, protection on benchmark inputs may not persist across alternative expressions of the same harmful request.
By Xinwei Zhang · arxiv.org
A short excerpt; the article and its images belong to the publisher. Summary, key facts and assessment above are AI-written from the article and can be wrong.