Pathology-knowledge Enhanced Multi-instance Prompt Learning for Few-shot Whole Slide Image Classification
Preprint
Abstract
Current multi-instance learning algorithms for pathology image analysis often require a substantial number of Whole Slide Images for effective training but exhibit suboptimal performance in scenarios with limited learning data. In clinical settings, restricted access to pathology slides is inevitable due to patient privacy concerns and the prevalence of rare or emerging diseases. The emergence of the Few-shot Weakly Supervised WSI Classification accommodates the significant challenge of the limited slide data and sparse slide-level labels for diagnosis. Prompt learning based on the pre-trained models (\eg, CLIP) appears to be a promising scheme for this setting; however, current research in this area is limited, and existing algorithms often focus solely on patch-level prompts or confine themselves to language prompts. This paper proposes a multi-instance prompt learning framework enhanced with pathology knowledge, \ie, integrating visual and textual prior knowledge into prompts at both patch and slide levels. The training process employs a combination of static and learnable prompts, effectively guiding the activation of pre-trained models and further facilitating the diagnosis of key pathology patterns. Lightweight Messenger (self-attention) and Summary (attention-pooling) layers are introduced to model relationships between patches and slides within the same patient data. Additionally, alignment-wise contrastive losses ensure the feature-level alignment between visual and textual learnable prompts for both patches and slides. Our method demonstrates superior performance in three challenging clinical tasks, significantly outperforming comparative few-shot methods.Keywords
All Related Versions
This publication has 12 references indexed in Scilit:
- Artificial intelligence for digital and computational pathologyNature Reviews Bioengineering, 2023
- A visual–language foundation model for pathology image analysis using medical TwitterNature Medicine, 2023
- DGMIL: Distribution Guided Multiple Instance Learning for Whole Slide Image Classification2022
- Learning to Prompt for Vision-Language ModelsInternational Journal of Computer Vision, 2022
- DT-MIL: Deformable Transformer for Multi-instance Learning on Histopathological Image2021
- Data-efficient and weakly supervised computational pathology on whole-slide imagesNature Biomedical Engineering, 2021
- Whole slide images based cancer survival prediction using attention guided deep multiple instance learning networksMedical Image Analysis, 2020
- Multiple Instance Learning with Center Embeddings for Histopathology Classification2020
- Clinical-grade computational pathology using weakly supervised deep learning on whole slide imagesNature Medicine, 2019
- Not-so-supervised: A survey of semi-supervised, multi-instance, and transfer learning in medical image analysisMedical Image Analysis, 2019