HRIL: Learning Multimodal Synergy via Higher-Order Tensor Modeling
HRIL models higher-order cross-modal tensors to capture synergistic multimodal information.
HRIL targets synergistic multimodal signals that appear only from joint modality configurations and cannot be recovered from any single modality. It builds an empirical cross-moment tensor over embeddings and uses Tucker decomposition plus a synergy-aware regularizer to preserve higher-order coupling. Experiments on a controlled synergy task and real-world benchmarks show consistent gains over multimodal contrastive methods, especially on synergy-dominated tasks. Code is released on GitHub.
- Models synergy via higher-order cross-moment tensors
- Tucker decomposition plus a synergy-aware regularizer
- Gains over contrastive methods on synergy-heavy tasks
- Code released at brightest66/HRIL
Full article179 words · extracted from arxiv.org · click to collapse
Self-supervised multimodal representation learning has achieved remarkable success across diverse domains, yet capturing synergistic information remains challenging due to the complexity of cross-modal interactions. Unlike the shared information across individual modalities, synergy arises when task-relevant signals emerge only from the joint configuration of multiple modalities and cannot be recovered from any modality in isolation. This work focuses on how to preserve the information capacity for such synergistic signals in multimodal representations. The key observation is that synergistic information is reflected in higher-order statistical dependence among modalities, which provides a principled target for explicitly modeling joint interactions. Motivated by this insight, we propose Higher-order Representation and Information Learning (HRIL), which constructs an empirical cross-moment tensor over modality embeddings to represent multi-way interactions. HRIL employs Tucker decomposition to obtain a core tensor, complemented by a synergy-aware regularizer that prevents energy concentration and preserves higher-order coupling capacity for synergistic information capture. Experiments on the controlled synergy task and real-world benchmarks demonstrate consistent improvements over existing multimodal contrastive methods, with notable gains on tasks dominated by synergistic interactions. Code is released at https://github.com/brightest66/HRIL.
Text extracted automatically; images, tables and formatting may be missing. Original: https://arxiv.org/abs/2610.12393