MTMedFormer: multi-task vision transformer for medical imaging with federated learning
Authors:
- 8 July 2025
- journal article
- Published by Springer Nature in Medical & Biological Engineering & Computing
- Vol. 63 (11) , 3421-3434
Abstract
Deep learning has revolutionized medical imaging, improving tasks like image segmentation, detection, and classification, often surpassing human accuracy. However, the training of effective diagnostic models is hindered by two major challenges: the need for large datasets for each task and privacy laws restricting the sharing of medical data. Multi-task learning (MTL) addresses the first challenge by enabling a single model to perform multiple tasks, though convolution-based MTL models struggle with contextualizing global features. Federated learning (FL) helps overcome the second challenge by allowing models to train collaboratively without sharing data, but traditional methods struggle to aggregate stable feature maps due to the permutation-invariant nature of neural networks. To tackle these issues, we propose MTMedFormer, a transformer-based multi-task medical imaging model. We leverage the transformers' ability to learn task-agnostic features using a shared encoder and utilize task-specific decoders for robust feature extraction. By combining MTL with a hybrid loss function, MTMedFormer learns distinct diagnostic tasks in a synergistic manner. Additionally, we introduce a novel Bayesian federation method for aggregating multi-task imaging models. Our results show that MTMedFormer outperforms traditional single-task and MTL models on mammogram and pneumonia datasets, while our Bayesian federation method surpasses traditional methods in image segmentation.Keywords
This publication has 39 references indexed in Scilit:
- One model is all you need: Multi-task learning enables simultaneous histology image segmentation and classificationMedical Image Analysis, 2022
- SFOD-Trans: semi-supervised fine-grained object detection framework with transformer moduleMedical & Biological Engineering & Computing, 2022
- Federated learning for predicting clinical outcomes in patients with COVID-19Nature Medicine, 2021
- Multi-task learning for segmentation and classification of tumors in 3D automated breast ultrasound imagesMedical Image Analysis, 2021
- Artificial intelligence, machine learning, computer-aided diagnosis, and radiomics: advances in imaging towards to precision medicineRadiologia Brasileira, 2019
- Combo loss: Handling input and output imbalance in multi-organ segmentationComputerized Medical Imaging and Graphics, 2019
- Developing and Validating a Survival Prediction Model for NSCLC Patients Through Distributed Learning Across 3 CountriesInternational Journal of Radiation Oncology - Biology - Physics (IJROBP), 2017
- INbreastAcademic Radiology, 2012
- Histopathological Image Analysis: A ReviewIEEE Reviews in Biomedical Engineering, 2009
- BI-RADS Classification for Management of Abnormal MammogramsThe Journal of the American Board of Family Medicine, 2006