Devils in Question Relay: Source-Conditioned Relay Steering to Mitigate Hallucinations in Audio-visual Large Language Models
SECRET, a training-free steering method, cuts source-confused hallucinations in audio-visual language models.
The paper analyzes source-confused grounding hallucinations in audio-visual large language models, where unused-modality cues drive answers the required modality does not support. Path interventions show question states relay interfering evidence, and cutting those pathways recovers correct-answer logits more than intervening at the generation position. SECRET, a training-free steering method, consistently beats prior training-free baselines on CMM and AVHBench across three AVLLMs, with gains up to 18.0 and 7.1 percentage points, and also helps open-ended captioning.
- AVLLMs suffer source-confused grounding when unused modalities inject unsupported cues.
- Analyses identify a question-relay path carrying interfering cross-modal evidence.
- Training-free SECRET steers question states toward the required source.
- Gains reach +18.0 and +7.1 points on CMM and AVHBench over base models.
From the original article
1 min read at huggingface.co
Audio-visual large language models (AVLLMs) have made remarkable progress in multimodal understanding and reasoning through interactions among visual, auditory, and linguistic information.
By Yu Zhang, Pingrui Zhang, Xuefeng Bai · huggingface.co
A short excerpt; the article and its images belong to the publisher. Summary, key facts and assessment above are AI-written from the article and can be wrong.