MFMF: Multiple Foundation Model Fusion Networks for Whole Slide Image Classification
Open Access
- 22 November 2024
- conference paper
- Published by Association for Computing Machinery (ACM)
- p. 1-8
Abstract
Tumor detection and subtyping remain a significant challenge in histopathology image analysis. As digital pathology progresses, the applications of deep learning become essential. Whole Slide Image (WSI) classification has emerged as a crucial task in digital pathology, vital for accurate cancer diagnosis and treatment. In this paper, we introduce an innovative abnormal-guided Multiple Foundation Model Fusion (MFMF) framework, aimed at enhancing WSI classification by integrating multi-level information from pathology images with Multiple Instance Learning (MIL). Traditional methods often focus on patch-level features while neglecting the rich contextual and morphological details at the cell and text levels, thus failing to fully exploit the multidimensional nature of WSIs. Our method enhances traditional models by efficiently integrating patch-level, cell-level, and text-level features using three foundation models. These are then fused through a novel three-step cross-attention module that effectively leverages cell and text information with patch-level features. Furthermore, unlike most studies that use attention scores to select instances based on the assumption that high scores indicate the presence of a tumor, we design an abnormality-aware module to naturally identify and detect abnormal features (i.e., tumors) as the criteria for selecting important instances, thereby reducing computational costs and boosting overall performance. We validate our approach against leading benchmarks on the CAMELYON16 and TCGA-Lung datasets, achieving superior classification performance. Our study not only tackles the challenges of sparsity and noise in multi-level features but also enhances the efficiency and accuracy of WSI classification by exploiting abnormal features.Keywords
Funding Information
- US National Science Foundation (IIS-2412195, CCF-2400785)
- Cancer Prevention and Research Institute of Texas (CPRIT) (RP230363)
This publication has 16 references indexed in Scilit:
- A visual-language foundation model for computational pathologyNature Medicine, 2024
- Artificial intelligence for digital and computational pathologyNature Reviews Bioengineering, 2023
- Cross-Modal Translation and Alignment for Survival Analysis2023
- Multi-scale representation attention based deep multiple instance learning for gigapixel whole slide image analysisMedical Image Analysis, 2023
- Scaling Vision Transformers to Gigapixel Images via Hierarchical Self-Supervised Learning2022
- Dual-stream Multiple Instance Learning Network for Whole Slide Image Classification with Self-supervised Contrastive LearningIEEE Computer Society Conference on Computer Vision and Pattern Recognition Workshops (CVPRW), 2021
- Artificial intelligence in cancer research, diagnosis and therapyNature Reviews Cancer, 2021
- Whole slide images based cancer survival prediction using attention guided deep multiple instance learning networksMedical Image Analysis, 2020
- Diagnostic Assessment of Deep Learning Algorithms for Detection of Lymph Node Metastases in Women With Breast CancerJAMA, 2017
- A method for normalizing histology slides for quantitative analysis2009