Lluna: Collaborative Unimodal Models for Missing-Modality Inference
Lluna improves missing-modality prediction by pairing private unimodal paths with graph collaboration, gaining 4.26% Mean-Missing.
Lluna targets cases where collaborating pretrained unimodal models become less accurate than a surviving unimodal model once inputs are missing. It keeps a direct private prediction path from adapted unimodal features and adds a graph-based collaboration branch as a residual correction, trained with Progressive Collaboration Consistency. Across eight datasets in affective computing, scene understanding, and medical prediction, it had the best average performance on non-full inputs, improving Mean-Missing by 4.26% versus the strongest evaluated baseline. Fixed-anchor tests showed positive mean collaboration gains under minimal inputs on all eight tasks. Code is published by AIFrontierLab.
- Private unimodal predictions are corrected by a shared graph collaboration branch.
- Progressive Collaboration Consistency aligns representations as modalities are removed.
- Mean-Missing improved 4.26% on average versus the strongest baseline.
- Positive collaboration gains held under minimal inputs on all eight tasks.
- Organizations
- AIFrontierLab
- AI models
- Lluna
From the original article
1 min read at huggingface.co
Pretrained unimodal models offer a flexible route to multimodal prediction through collaboration, but this collaboration can become less accurate than a surviving unimodal model when modalities are missing. We study this setting using fixed surviving-model anchors and find that collaborative gains can weaken or reverse as inputs disappear, even when the anchor's own input is unchanged.
By Haoyue Bai, Yinyi Luo, Heqing Huang · huggingface.co
A short excerpt; the article and its images belong to the publisher. Summary, key facts and assessment above are AI-written from the article and can be wrong.