VLM safety steering differs by layer depth, study finds
Original title: Depth-Dependent Safety Geometry in Vision-Language Models: Layer-Wise Localization and Bidirectional Refusal Control
On Qwen3-VL-4B, early vision-layer steering moved unsafe rejection between 6% and 96% without retraining.
Researchers describe an inference-time method that localizes and steers safety-sensitive representations in frozen vision-language models. On Qwen3-VL-4B, early vision layers allowed strong bidirectional refusal control, while later layers were more sensitive to activation magnitude and showed a safety-selectivity trade-off. Deeper layers stayed separable along the safety direction but largely resisted steering. InternVL2.5-4B showed a similar depth-dependent pattern. The authors frame this as a white-box refusal-bypass surface in VLM vision encoders.
- Unsafe rejection under early-layer steering spanned 6% to 96% on Qwen3-VL-4B.
- Later layers responded more to magnitude scaling than to direction steering.
- InternVL2.5-4B showed qualitatively similar geometry with different layer locations.
From the original article
1 min read at arxiv.orgVision-Language Models (VLMs) are widely deployed, yet the internal mechanisms behind their safety behavior remain poorly understood. We present a lightweight inference-time framework for localizing and manipulating safety-sensitive representations in frozen VLMs, without retraining or modifying parameters. On Qwen3-VL-4B, layer-wise activation divergence analysis reveals non-uniform safety geometry across depth.
By Muhammad Zeeshan Karamat · arxiv.org
A short excerpt; the article and its images belong to the publisher. Summary, key facts and assessment above are AI-written from the article and can be wrong.