FAMF-Net: Feature Alignment Mutual Attention Fusion With Region Awareness for Breast Cancer Diagnosis via Imbalanced Data
- 5 November 2024
- journal article
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Transactions on Medical Imaging
- Vol. 44 (3) , 1153-1167
Abstract
Automatic and accurate classification of breast cancer in multimodal ultrasound images is crucial to improve patients’ diagnosis and treatment effect and save medical resources. Methodologically, the fusion of multimodal ultrasound images often encounters challenges such as misalignment, limited utilization of complementary information, poor interpretability in feature fusion, and imbalances in sample categories. To solve these problems, we propose a feature alignment mutual attention fusion method (FAMF-Net), which consists of a region awareness alignment (RAA) block, a mutual attention fusion (MAF) block, and a reinforcement learning-based dynamic optimization strategy(RDO). Specifically, RAA achieves region awareness through class activation mapping and performs translation transformation to achieve feature alignment. When MAF utilizes a mutual attention mechanism for feature interaction fusion, it mines edge and color features separately in B-mode and shear wave elastography images, enhancing the complementarity of features and improving interpretability. Finally, RDO uses the distribution of samples and prediction probabilities during training as the state of reinforcement learning to dynamically optimize the weights of the loss function, thereby solving the problem of class imbalance. The experimental results based on our clinically obtained dataset demonstrate the effectiveness of the proposed method. Our code will be available at: https://github.com/Magnety/Multi_modal_Image.Keywords
Funding Information
- National Natural Science Foundation of China (62371312, U22A2024, 62271328, 62301329)
- National Natural Science Foundation of Guangdong Province (2022A1515110704)
- Post-Doctoral Science Foundation of China (2023M732358)
- Shenzhen Science and Technology Program (JCYJ20220818095809021, KCXFZ20201221173213036)
- Innovation and Technology Fund—Guangdong-Hong Kong Technology Cooperation Funding Scheme (GHP/050/20SZ)
- General Research Fund of Hong Kong Research Grants Council (15218521)
- Collaborative Research With World-Leading Research Groups of The Hong Kong Polytechnic University (P0052927)
This publication has 56 references indexed in Scilit:
- Deep learning‐based multimodal fusion network for segmentation and classification of breast cancers using B‐mode and elastography ultrasound imagesBioengineering & Translational Medicine, 2022
- Deep Multimodal Guidance for Medical Image Classification2022
- The importance of multi-modal imaging and clinical information for humans and AI-based algorithms to classify breast masses (INSPiRED 003): an international, multicenter analysisEuropean Radiology, 2022
- Auto-weighting for Breast Cancer Classification in Multimodal Ultrasound2020
- Deep reinforcement learning for imbalanced classificationApplied Intelligence, 2020
- ShuffleNet V2: Practical Guidelines for Efficient CNN Architecture Design2018
- Stiffness of tumours measured by shear-wave elastography correlated with subtypes of breast cancerEuropean Radiology, 2013
- Diagnostic value of commercially available shear-wave elastography for breast cancers: integration into BI-RADS classification with subcategories of category 4European Radiology, 2013
- Estimation of the Youden Index and its Associated Cutoff PointBiometrical Journal, 2005
- Gabor filter-based edge detectionPattern Recognition, 1992