From Routing Signals to Selective Review: Visual regrounding in MoE VLMs
MoE routing signals detect missing visual targets in Qwen and Gemma VLMs and trigger selective review.
The paper uses Mixture-of-Experts routing probabilities to detect target-absence grounding failures in vision-language models before generation. Linear detectors on Qwen3-VL-30B-A3B-Instruct and Gemma-4-26B-A4B-it reach ROC-AUCs of 0.9988 and 0.9956 on GQA-Inpaint, retaining 0.8095 and 0.7781 on OBER. A routing-gated review prompt raises end-to-end accuracy by up to 22.25% for Qwen and 13.42% for Gemma without modifying weights.
- Routing-only detectors exceed 0.99 ROC-AUC on GQA-Inpaint.
- Qwen accuracy rises 22.25% on GQA-Inpaint and 12.17% on OBER.
- Signal is early, target-token localized, and spread across experts.
- Selective review requires no model-weight changes.
Full article236 words · extracted from arxiv.org · click to collapse
Vision-language models (VLMs) may accept false visual premises, answering questions about a target object's color, count, location, or state even when it is absent. We call this reliability-critical behavior a target-absence grounding failure. Existing visual-grounding detectors primarily rely on generated responses, hidden states, or uncertainty measures. We present the first framework to leverage internal routing decisions in Mixture-of-Experts (MoE) VLMs to detect target absence before generation and guide selective correction. We extract target-token routing probabilities from Qwen3-VL-30B-A3B-Instruct and Gemma-4-26B-A4B-it, train a separate L2-regularized linear detector for each model, and use its predictions to selectively invoke a target-aware review prompt. Using routing alone, the Qwen and Gemma detectors achieve ROC-AUCs of 0.9988 and 0.9956 on GQA-Inpaint and retain 0.8095 and 0.7781 on the external OBER dataset, respectively. The resulting routing-gated policy improves end-to-end accuracy on GQA-Inpaint and OBER by +22.25% and +12.17% for Qwen, and by +13.42% and +1.39% for Gemma, without modifying model weights. Further analysis shows that the signal is localized to the target-object token, emerges in early MoE layers, and is distributed across partially substitutable experts. Although cross-dataset threshold shifts require recalibration, false-positive review causes limited harm overall, suggesting that intervention risk can be controlled through joint selection of the detector threshold and review prompt. Overall, we show that routing probabilities alone preserve actionable information about visual perception, allowing computation already produced by an MoE VLM to support low-cost detection and selective visual regrounding.
Text extracted automatically; images, tables and formatting may be missing. Original: https://arxiv.org/abs/2609.38111