Federated Foundation Model for Cardiac CT Imaging
Authors:
- Malte TölleMTMalte Tölle
- Moritz BednorzMBMoritz Bednorz
Preprint
Abstract
Federated learning (FL) is a renowned technique for utilizing decentralized data while preserving privacy. However, real-world applications often involve inherent challenges such as partially labeled datasets, where not all clients possess expert annotations of all labels of interest, leaving large portions of unlabeled data unused. In this study, we conduct the largest federated cardiac CT imaging analysis to date, focusing on partially labeled datasets ($n=8,124$) of Transcatheter Aortic Valve Implantation (TAVI) patients over eight hospital clients. Transformer architectures, which are the major building blocks of current foundation models, have shown superior performance when trained on larger cohorts than traditional CNNs. However, when trained on small task-specific labeled sample sizes, it is currently not feasible to exploit their underlying attention mechanism for improved performance. Therefore, we developed a two-stage semi-supervised learning strategy that distills knowledge from several task-specific CNNs (landmark detection and segmentation of calcification) into a single transformer model by utilizing large amounts of unlabeled data typically residing unused in hospitals to mitigate these issues. This method not only improves the predictive accuracy and generalizability of transformer-based architectures but also facilitates the simultaneous learning of all partial labels within a single transformer model across the federation. Additionally, we show that our transformer-based model extracts more meaningful features for further downstream tasks than the UNet-based one by only training the last layer to also solve segmentation of coronary arteries. We make the code and weights of the final model openly available, which can serve as a foundation model for further research in cardiac CT imaging.Keywords
All Related Versions
This publication has 48 references indexed in Scilit:
- FUNAvg: Federated Uncertainty Weighted Averaging for Datasets with Diverse Labels2024
- Towards Unified Multi-modal Dataset Creation for Deep Learning Utilizing Structured Reports2024
- ConDistFL: Conditional Distillation for Federated Learning from Partially Annotated Data2023
- Content-Aware Differential Privacy with Conditional Invertible Neural Networks2022
- Swin UNETR: Swin Transformers for Semantic Segmentation of Brain Tumors in MRI Images2022
- Impact of Left Ventricular Outflow Tract Calcification on Outcomes Following Transcatheter Aortic Valve ReplacementCardiovascular Revascularization Medicine, 2022
- Cascaded neural network-based CT image processing for aortic root analysisInternational Journal of Computer Assisted Radiology and Surgery, 2022
- TorchIO: A Python library for efficient loading, preprocessing, augmentation and patch-based sampling of medical images in deep learningComputer Methods and Programs in Biomedicine, 2021
- Fed-BioMed: A General Open-Source Frontend Framework for Federated Learning in Healthcare2020
- Generalised Dice Overlap as a Deep Learning Loss Function for Highly Unbalanced Segmentations2017