Tri-PvP: Exposing Modality Bias in Omni-Modal Large Language Models through Perceptual-Propositional Evidence Conflicts
Tri-PvP is a new benchmark revealing robust modality bias in omni-modal LLMs, with vision favoring perceptual signals and audio favoring propositional ones.
The paper introduces Tri-PvP, an 8,000-sample tri-modal conflict benchmark designed to evaluate modality bias in omni-modal large language models (OLLMs) that jointly process vision, audio, and text. The benchmark is designed to disentangle evidence-form bias from true modality bias by using both perceptual and propositional signals for vision and audio. Evaluating five OLLMs, the researchers found robust visual bias across most models and evidence-type conditions, along with a systematic asymmetry where models favor perceptual signals in vision but propositional signals in audio. Further analysis via layer-wise linear probing and contrastive decoding revealed that this modality bias is already linearly decodable from early representation layers and can only be partially mitigated.
- Tri-PvP is an 8,000-sample benchmark for evaluating modality bias in omni-modal LLMs.
- It crosses vision, audio, and text, using both perceptual and propositional evidence forms.
- Evaluation of five OLLMs reveals robust visual bias and a systematic asymmetry in evidence-form bias.
- Layer-wise probing shows modality bias is present in early representation layers and is hard to mitigate.
Full article168 words · extracted from huggingface.co · click to collapse
Omni-modal large language models (OLLMs) jointly process vision, audio, and text, yet their modality bias under cross-modal conflict remains underexplored. Existing benchmarks conflate two distinct forms of evidence within a single modality: perceptual signals (e.g., a photograph or recording of a dog) and propositional signals (e.g., the declarative claim "this is a dog"), such that any measured modality bias is inherently confounded with evidence-form bias, precluding clean attribution to either source. To address this, we introduce Tri-PvP, an 8,000-sample tri-modal conflict benchmark crossing vision, audio, and text, where vision and audio each take perceptual or propositional form. Evaluating five OLLMs, we find robust visual bias across most models and evidence-type conditions. Crucially, we reveal a systematic asymmetry in evidence-form bias: models exhibit a stronger bias toward perceptual signal in vision but propositional in audio. Further analyses via layer-wise linear probing and contrastive decoding reveal that modality bias is already linearly decodable from early representation layers and can only be partially mitigated, calling for mitigation strategies beyond surface-level interventions.
Text extracted automatically; images, tables and formatting may be missing. Original: https://huggingface.co/papers/2609.06011