Buffalo: Biomedical Vision-Language Understanding with Cross-Modal Prototype and Federated Foundation Model Collaboration
Open Access
- 21 October 2024
- conference paper
- Published by Association for Computing Machinery (ACM)
- p. 2775-2785
Abstract
Federated learning (FL) enables collaborative learning across multiple biomedical data silos with multimodal foundation models while preserving privacy. Due to the heterogeneity in data processing and collection methodologies across diverse medical institutions and the varying medical inspections patients undergo, modal heterogeneity exists in practical scenarios, where severe modal heterogeneity may even prevent model training. With privacy considerations, data transfer cannot be permitted, restricting knowledge exchange among different clients. To trickle these issues, we propose a cross-modal prototype imputation method for visual-language understanding (Buffalo) with only a slight increase in communication cost, which can improve the performance of fine-tuning general foundation models for downstream biomedical tasks. We conducted extensive experiments on medical report generation and biomedical visual question-answering tasks. The results demonstrate that Buffalo can fully utilize data from all clients to improve model generalization compared to other modal imputation methods in three modal heterogeneity scenarios, approaching or even surpassing the performance in the ideal scenario without missing modality.Keywords
Funding Information
- Hunan Provincial Natural Science Foundation of China (No.2023JJ70008)
- the National Key Research and Development Plan of China (No.2023YFC3604802)
- the youth innovation promotion association cas
- The science and technology innovation Program of Hunan Province (No.2022RC4006, No.2023WK2005)
- the Postdoctoral Fellowship Program of CPSF (No.GZC20232737)
This publication has 29 references indexed in Scilit:
- Federated Learning on Multimodal Data: A Comprehensive SurveyMachine Intelligence Research, 2023
- Towards artificial general intelligence via a multimodal foundation modelNature Communications, 2022
- Federated Learning on Non-IID Data Silos: An Experimental Study2022
- Multimodal Federated Learning on IoT Data2022
- Multi-institutional Collaborations for Improving Deep Learning-based Magnetic Resonance Image Reconstruction Using Federated LearningConference on Computer Vision and Pattern Recognition (CVPR), 2021
- Swin Transformer: Hierarchical Vision Transformer using Shifted Windows2021
- The future of digital health with federated learningnpj Digital Medicine, 2020
- Multi-view Deep Network for Cross-View Classification2016
- Preparing a collection of radiology examinations for distribution and retrievalJournal of the American Medical Informatics Association, 2015
- CIDEr: Consensus-based image description evaluation2015