An Alignment and Imputation Network (AINet) for Breast Cancer Diagnosis With Multimodal Multi-View Ultrasound Images
- 24 October 2025
- journal article
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Transactions on Medical Imaging
- Vol. 45 (4) , 1383-1394
Abstract
Recently, numerous deep learning models have been proposed for breast cancer diagnosis using multimodal multi-view ultrasound images. However, their performance could be highly affected by overlooking interactions between different modalities and views. Moreover, existing methods struggle to handle cases where certain modalities or views are missing, which limits their clinical applications. To address these issues, we propose a novel Alignment and Imputation Network (AINet) by integrating 1) alignment and imputation pre-training, and 2) hierarchical fusion fine-tuning. Specifically, in the pre-training stage, cross-modal contrastive learning is employed to align features across different modalities, for effectively capturing inter-modal interactions. To simulate missing modality (view) scenarios, we randomly mask out features and then impute them by leveraging inter-modal and inter-view relationships. Following the clinical diagnosis procedure, the subsequent fine-tuning stage further incorporates modality-level and view-level fusion in a hierarchical manner. The proposed AINet is developed and evaluated on three datasets, comprising 15,223 subjects in total. Experimental results demonstrate that AINet significantly outperforms state-of-the-art methods, particularly in handling missing modalities (views). This highlights its robustness and potential for real-world clinical applications.Keywords
Funding Information
- National Natural Science Foundation of China (U23A20295)
- National Natural Science Foundation of China (62131015)
- National Natural Science Foundation of China (82441023)
- National Natural Science Foundation of China (82202160)
- China Ministry of Science and Technology (STI2030-Major Projects-2022ZD0209000)
- China Ministry of Science and Technology (STI2030-Major Projects-2022ZD0213100)
- Shanghai Municipal Central Guided Local Science and Technology Development Fund (YDZX20233100001001)
- Key Research and Development Program of Guangdong Province, China (2023B0303040001)
- ShanghaiTech University
This publication has 50 references indexed in Scilit:
- IRUM: An Image Representation and Unified Learning Method for Breast Cancer Diagnosis from Multi-View Ultrasound Images2024
- Graph neural network-based breast cancer diagnosis using ultrasound images with optimized graph construction integrating the medically significant featuresJournal of Cancer Research and Clinical Oncology, 2023
- MoSID: Modality-Specific Information Disentanglement from Multi-parametric MRI for Breast Tumor Segmentation2023
- Breast Ultrasound Tumor Classification Using a Hybrid Multitask CNN-Transformer Network2023
- CXR-CLIP: Toward Large Scale Chest X-ray Language-Image Pre-training2023
- HAU-Net: Hybrid CNN-transformer for breast ultrasound image segmentationBiomedical Signal Processing and Control, 2023
- Personalized Diagnostic Tool for Thyroid Cancer Classification Using Multi-view Ultrasound2022
- A VGG attention vision transformer network for benign and malignant classification of breast ultrasound imagesMedical Physics, 2022
- Deep learning-based breast cancer classification through medical imaging modalities: state of the art and research challengesArtificial Intelligence Review, 2019
- ImageNet Large Scale Visual Recognition ChallengeInternational Journal of Computer Vision, 2015