SemMSA: Latent Semantic-Aided Robust Multimodal Sentiment Analysis with Incomplete Data
SemMSA reaches state-of-the-art multimodal sentiment analysis on incomplete data using latent LLM semantics.
SemMSA is a multimodal sentiment-analysis framework for incomplete language, visual, and acoustic inputs. Cross-modal Semantic Refinement uses adapters and a frozen LLM to iteratively produce continuous semantic states without decoding text, and Cross-modal Spectral Alignment matches those semantics to every modality via the dominant component of a kernel Gram matrix. An instance-level spectral separation constraint is added to preserve discriminability. Experiments on SIMS, MOSI, and MOSEI are reported as state of the art.
- SemMSA builds sentiment semantics in a frozen LLM without decoding text.
- Spectral alignment matches those semantics to language, vision, and audio.
- An instance-level spectral constraint is used to limit representation collapse.
- The method reports state-of-the-art results on SIMS, MOSI, and MOSEI.
From the original article
1 min read at arxiv.orgRecent research on Multimodal Sentiment Analysis (MSA) has focused on learning from language, visual, and acoustic modalities with incomplete data to infer human sentiment. Most studies typically compensate for missing information by reconstructing modality features or designing complicated fusion mechanisms.
By Wenhao Li · arxiv.org
A short excerpt; the article and its images belong to the publisher. Summary, key facts and assessment above are AI-written from the article and can be wrong.