A Federated Multimodal Co-Attention Framework for Privacy-Preserving Healthcare Analytics

Authors:

Abstract
Accurate, interpretable, and privacy-preserving diagnosis of thoracic diseases remains a criticalchallenge for healthcare organizations seeking to improve clinical decision-making whileprotecting patient data. Conventional deep learning models that rely on a single data modality,such as chest radiographs or radiology reports, often fail to capture complementary clinicalinformation and provide limited interpretability. Although multimodal artificial intelligence(AI) approaches have demonstrated improved diagnostic performance by integrating visualand textual information, most require centralized data aggregation, creating concerns regardingpatient privacy, regulatory compliance, and deployment across distributed healthcare environments.This study proposes FedCoAttention, a federated multimodal co-attention framework forhealthcare analytics and clinical decision support. The framework integrates pretrained DistilledBidirectional Encoder Representations from Transformers (DistilBERT) and Residual Network-50 (ResNet50) encoders through a transformer-based co-attention mechanism that captures bidirectionalinteractions between textual tokens and image patches. Collaborative model trainingis achieved using Federated Averaging (FedAvg), enabling multiple healthcare institutions tojointly develop predictive models without sharing raw patient data. To support transparency andclinician trust, the framework incorporates Patch Importance Maps and Gradient-weighted ClassActivation Mapping (Grad-CAM) visualizations that align clinically relevant report contentwith salient radiographic regions. Experimental evaluation on the Indiana Chest X-ray dataset,consisting of Normal, Cardiomegaly, and Atelectasis cases, demonstrates strong predictiveperformance and stable federated convergence. The proposed framework achieved a best globalaccuracy of 98.2%, a macro F1-score of 0.96, and a weighted F1-score of 0.98. Across threeindependent federated training runs, the model attained a mean accuracy of 97.7% ± 0.4%,indicating robust and reproducible performance. The findings demonstrate the potential offederated multimodal analytics to support privacy-preserving clinical decision support acrossdistributed healthcare organizations while enhancing diagnostic accuracy, interpretability, andscalability.