OmniMed-FL: A Robust Multimodal Federated Learning Framework for Clinical Diagnosis
OmniMed-FL benchmarks multimodal federated learning for chest radiograph diagnosis across 3-20 clients, with FedProx leading under severe non-IID skew.
OmniMed-FL studies multimodal federated learning combining chest radiographs and clinical notes for five-class condition classification under HIPAA/GDDR-compliant decentralized training. It benchmarks eight fusion strategies, imputation rules, and federated baselines under Dirichlet non-IID partitioning across 3-20 hospital clients. With 5 clients and severe skew (alpha=0.1), FedProx scored 0.737 macro-F1 versus 0.662 for FedAvg and 0.297 for local-only training. Multimodal fusion beat unimodal inputs (0.956 vs 0.934 text, 0.664 images) on the synthetic corpus.
- Controlled study of image+text federated learning across 3-20 clients
- FedProx (0.737 F1) outperforms FedAvg (0.662) and local-only (0.297) under skew
- Label skew costs up to 0.27 F1, more than sevenfold client increase (0.10)
- Descriptive proxy comparisons; notes are synthetic, not patient-level
Full article262 words · extracted from arxiv.org · click to collapse
Simultaneous assessment of medical imaging and patient records is often required in clinical diagnosis. However, standard machine learning algorithms cannot analyze these data types together. Meanwhile, compliance with HIPAA and GDPR can constrain centralized aggregation of sensitive patient data. This leaves a crucial void of secure fusion of visual and textual context across distant networks. Thus, we present OmniMed-FL, a controlled systems study of multimodal federated learning for five-class clinical condition classification (Normal, Pneumonia, COVID-19, Pleural Effusion, Cardiomegaly). Our proxy corpus pairs 3,000 public chest radiographs with 3,000 class-conditioned synthetic notes, matched by class, not by patient. The framework benchmarks eight fusion strategies, three initializations, four missing-text imputation rules, and matched federated baselines under non-IID Dirichlet partitioning across 3 to 20 hospital clients. As all notes are synthetic and pairing is not patient-level, these are descriptive proxy comparisons, not estimates of diagnostic performance or deployment readiness. Within those limits with clients ($K=5$) and severe skew ($α=0.1$), local-only training achieves a macro-F1 score of 0.297, FedAvg achieves $0.662\pm0.074$, FedProx $0.737\pm0.085$, a matched FedMME-style one-shot ensemble $0.647\pm0.080$, and our SCAFFOLD-AdamW adaptation $0.070\pm0.015$, the 0.075 FedProx-FedAvg gap falling inside the wider of the two two-seed standard deviations. Over a $4\times3$ grid, label skew costs up to 0.27 F1 whereas a near-sevenfold client increase costs at most 0.10, while bidirectional volume grows linearly to 183.5 GiB at $K=20$. Multimodal fusion leads on both corpora, scoring 0.956 against 0.934 for text and 0.664 for images on the synthetic corpus and 0.906 against 0.880 and 0.737 on the radiograph corpus, for $2.3\times$ the model state of text alone.
Text extracted automatically; images, tables and formatting may be missing. Original: https://arxiv.org/abs/2609.10364