Abstract
Recently, numerous deep learning models have been proposed for breast cancer diagnosis using multimodal multi-view ultrasound images. However, their performance could be highly affected by overlooking interactions between different modalities and views. Moreover, existing methods struggle to handle cases where certain modalities or views are missing, which limits their clinical applications. To address these issues, we propose a novel Alignment and Imputation Network (AINet) by integrating 1) alignment and imputation pre-training, and 2) hierarchical fusion fine-tuning. Specifically, in the pre-training stage, cross-modal contrastive learning is employed to align features across different modalities, for effectively capturing inter-modal interactions. To simulate missing modality (view) scenarios, we randomly mask out features and then impute them by leveraging inter-modal and inter-view relationships. Following the clinical diagnosis procedure, the subsequent fine-tuning stage further incorporates modality-level and view-level fusion in a hierarchical manner. The proposed AINet is developed and evaluated on three datasets, comprising 15,223 subjects in total. Experimental results demonstrate that AINet significantly outperforms state-of-the-art methods, particularly in handling missing modalities (views). This highlights its robustness and potential for real-world clinical applications.
Funding Information
  • National Natural Science Foundation of China (U23A20295)
  • National Natural Science Foundation of China (62131015)
  • National Natural Science Foundation of China (82441023)
  • National Natural Science Foundation of China (82202160)
  • China Ministry of Science and Technology (STI2030-Major Projects-2022ZD0209000)
  • China Ministry of Science and Technology (STI2030-Major Projects-2022ZD0213100)
  • Shanghai Municipal Central Guided Local Science and Technology Development Fund (YDZX20233100001001)
  • Key Research and Development Program of Guangdong Province, China (2023B0303040001)
  • ShanghaiTech University

This publication has 50 references indexed in Scilit: