Abstract
Federated learning (FL) enables collaborative learning across multiple biomedical data silos with multimodal foundation models while preserving privacy. Due to the heterogeneity in data processing and collection methodologies across diverse medical institutions and the varying medical inspections patients undergo, modal heterogeneity exists in practical scenarios, where severe modal heterogeneity may even prevent model training. With privacy considerations, data transfer cannot be permitted, restricting knowledge exchange among different clients. To trickle these issues, we propose a cross-modal prototype imputation method for visual-language understanding (Buffalo) with only a slight increase in communication cost, which can improve the performance of fine-tuning general foundation models for downstream biomedical tasks. We conducted extensive experiments on medical report generation and biomedical visual question-answering tasks. The results demonstrate that Buffalo can fully utilize data from all clients to improve model generalization compared to other modal imputation methods in three modal heterogeneity scenarios, approaching or even surpassing the performance in the ideal scenario without missing modality.
Funding Information
  • Hunan Provincial Natural Science Foundation of China (No.2023JJ70008)
  • the National Key Research and Development Plan of China (No.2023YFC3604802)
  • the youth innovation promotion association cas
  • The science and technology innovation Program of Hunan Province (No.2022RC4006, No.2023WK2005)
  • the Postdoctoral Fellowship Program of CPSF (No.GZC20232737)

This publication has 29 references indexed in Scilit: