Tri-PvP: Exposing Modality Bias in Omni-Modal Large Language Models through Perceptual-Propositional Evidence Conflicts
Tri-PvP is a new benchmark revealing robust modality bias in omni-modal LLMs, with vision favoring perceptual signals and audio favoring propositional ones.
The paper introduces Tri-PvP, an 8,000-sample tri-modal conflict benchmark designed to evaluate modality bias in omni-modal large language models (OLLMs) that jointly process vision, audio, and text. The benchmark is designed to disentangle evidence-form bias from true modality bias by using both perceptual and propositional signals for vision and audio. Evaluating five OLLMs, the researchers found robust visual bias across most models and evidence-type conditions, along with a systematic asymmetry where models favor perceptual signals in vision but propositional signals in audio. Further analysis via layer-wise linear probing and contrastive decoding revealed that this modality bias is already linearly decodable from early representation layers and can only be partially mitigated.
- Tri-PvP is an 8,000-sample benchmark for evaluating modality bias in omni-modal LLMs.
- It crosses vision, audio, and text, using both perceptual and propositional evidence forms.
- Evaluation of five OLLMs reveals robust visual bias and a systematic asymmetry in evidence-form bias.
- Layer-wise probing shows modality bias is present in early representation layers and is hard to mitigate.
From the original article
1 min read at huggingface.co
Omni-modal large language models (OLLMs) jointly process vision, audio, and text, yet their modality bias under cross-modal conflict remains underexplored. Existing benchmarks conflate two distinct forms of evidence within a single modality: perceptual signals (e. g.
By Yen-Ting Piao, Shu-Yun Chen, Chin-Hui Chu · huggingface.co
A short excerpt; the article and its images belong to the publisher. Summary, key facts and assessment above are AI-written from the article and can be wrong.