Unleash the Power of State Space Model for Whole Slide Image With Local Aware Scanning and Importance Resampling
- 7 October 2024
- journal article
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Transactions on Medical Imaging
- Vol. 44 (2) , 1032-1042
Abstract
Whole slide image (WSI) analysis is gaining prominence within the medical imaging field. However, previous methods often fall short of efficiently processing entire WSIs due to their gigapixel size. Inspired by recent developments in state space models, this paper introduces a new Pathology Mamba (PAM) for more accurate and robust WSI analysis. PAM includes three carefully designed components to tackle the challenges of enormous image size, the utilization of local and hierarchical information, and the mismatch between the feature distributions of training and testing during WSI analysis. Specifically, we design a Bi-directional Mamba Encoder to process the extensive patches present in WSIs effectively and efficiently, which can handle large-scale pathological images while achieving high performance and accuracy. To further harness the local information and inherent hierarchical structure of WSI, we introduce a novel Local-aware Scanning module, which employs a local-aware mechanism alongside hierarchical scanning to adeptly capture both the local information and the overarching structure within WSIs. Moreover, to alleviate the patch feature distribution misalignment between training and testing, we propose a Test-time Importance Resampling module to conduct testing patch resampling to ensure consistency of feature distribution between the training and testing phases, and thus enhance model prediction. Extensive evaluation on nine WSI datasets with cancer subtyping and survival prediction tasks demonstrates that PAM outperforms current state-of-the-art methods and also its enhanced capability in modeling discriminative areas within WSIs. The source code is available at https://github.com/HKU-MedAI/PAM.Keywords
Funding Information
- Research Grants Council of Hong Kong (27206123, T45-401/22-N)
- Innovation and Technology Commission (ITS/274/22, ITS/273/22)
- National Natural Science Foundation of China (62201483)
- Guangdong Natural Science Fund (2024A1515011875)
This publication has 38 references indexed in Scilit:
- MambaMIL: Enhancing Long Sequence Modeling with Sequence Reordering in Computational Pathology2024
- Structured State Space Models for Multiple Instance Learning in Digital Pathology2023
- OTFPF: Optimal transport based feature pyramid fusion network for brain age estimationInformation Fusion, 2023
- Transformer-based unsupervised contrastive learning for histopathological image classificationMedical Image Analysis, 2022
- Whole Slide Images are 2D Point Clouds: Context-Aware Survival Prediction Using Patch-Based Graph Convolutional Networks2021
- HACT-Net: A Hierarchical Cell-to-Tissue Graph Neural Network for Histopathological Image Classification2020
- Hover-Net: Simultaneous segmentation and classification of nuclei in multi-tissue histology imagesMedical Image Analysis, 2019
- Sigmoid-weighted linear units for neural network function approximation in reinforcement learningNeural Networks, 2018
- Diagnostic Assessment of Deep Learning Algorithms for Detection of Lymph Node Metastases in Women With Breast CancerJAMA, 2017
- Report of two cases of pseudoprogression in patients with non–small cell lung cancer treated with nivolumab—including histological analysis of one case after tumor regressionLung Cancer, 2016