MLLMs Hallucinate when Information Distribution Drifts in Synergy Heads
Researchers propose HEAL, a method that reduces multimodal LLM hallucinations by calibrating information distribution drift in synergy attention heads.
A paper introduces HEAL (Head-lEvel information disentAnglement and caLibration), a technique to identify and mitigate hallucinations in multimodal large language models. It uses causal noise intervention and counterfactual Difference-in-Differences to disentangle head information distribution, finding hallucinations occur when distribution drifts from equilibrium in synergy heads. HEAL injects dynamic calibration factors into synergy head value vectors to steer outputs toward factual evidence, reducing hallucinations across multiple MLLMs.
- HEAL filters causally redundant heads via causal noise intervention on multi-head outputs.
- Counterfactual Difference-in-Differences categorizes heads into four information types.
- Hallucinations tied to distribution drift in synergy heads, not modality-specific head counts.
- Dynamic calibration factors injected into synergy head value vectors reduce hallucinations.
Full article167 words · extracted from huggingface.co · click to collapse
Multimodal Large Language Models (MLLMs) often struggle with hallucinations, thus hindering their reliable practical applications. Existing attention-based mitigation methods mainly rely on indirect signals (e.g., attention weights) that fail to accurately reflect the actual information shift underlying hallucination generation. In this paper, we propose HEAL, Head-lEvel information disentAnglement and caLibration for identifying and mitigating hallucinations. HEAL first employs causal noise intervention on multi-head outputs to filter out causally redundant heads. Subsequently, it disentangles information distribution within the remaining heads via the counterfactual Difference-in-Differences, categorizing heads into four types. Through analysis, we observe: hallucinations happen when information distribution drifts away from a healthy equilibrium in synergy heads, not strongly correlated with the quantity or strength of modality-specific heads. Motivated by this insight, HEAL injects dynamic information calibration factors into the value vectors of synergy heads, and actively regulates visual-language dependencies, steering the output distribution towards factual evidence. Extensive experiments demonstrate that HEAL effectively reduces hallucinations across multiple MLLMs, offering a simple and interpretable pathway to enhance model trustworthiness.
Text extracted automatically; images, tables and formatting may be missing. Original: https://huggingface.co/papers/2609.09206