Prompting Vision-Language Model for Nuclei Instance Segmentation and Classification
- 25 June 2025
- journal article
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Transactions on Medical Imaging
- Vol. 44 (11) , 4567-4578
Abstract
Nuclei instance segmentation and classification are a fundamental and challenging task in whole slide Imaging (WSI) analysis. Most dense nuclei prediction studies rely heavily on crowd labelled data on high-resolution digital images, leading to a time-consuming and expertise-required paradigm. Recently, Vision-Language Models (VLMs) have been intensively investigated, which learn rich cross-modal correlation from large-scale image-text pairs without tedious annotations. Inspired by this, we build a novel framework, called PromptNu, aiming at infusing abundant nuclei knowledge into the training of the nuclei instance recognition model through vision-language contrastive learning and prompt engineering techniques. Specifically, our approach starts with the creation of multifaceted prompts that integrate comprehensive nuclear knowledge, including visual insights from the GPT-4V model, statistical analyses, and expert insights from the pathology field. Then, we propose a novel prompting methodology that consists of two pivotal vision-language contrastive learning components: the Prompting Nuclei Representation Learning (PNuRL) and the Prompting Nuclei Dense Prediction (PNuDP), which adeptly integrates the expertise embedded in pre-trained VLMs and multi-faceted prompts into the feature extraction and prediction process, respectively. Comprehensive experiments on six datasets with extensive WSI scenarios demonstrate the effectiveness of our method for both nuclei instance segmentation and classification tasks. The code is available at https://github.com/NucleiDet/PromptNu.Keywords
Funding Information
- National Natural Science Foundation of China (62322605, 62272468, U24A20341)
- Project of Key Industrial Innovation Chain (Cluster) in Shaanxi Province—Social Development Field (2023-ZDLSF-41)
- Anhui Provincial Key Research and Development Programmes (2023s07020001)
This publication has 60 references indexed in Scilit:
- Prompting Whole Slide Image Based Genetic Biomarker Prediction2024
- Low-Shot Prompt Tuning for Multiple Instance Learning Based Histology Classification2024
- CLIP-Lung: Textual Knowledge-Guided Lung Nodule Malignancy Prediction2023
- DiffMix: Diffusion Model-Based Data Synthesis for Nuclei Segmentation and Classification in Imbalanced Pathology Image Datasets2023
- Diffusion-Based Data Augmentation for Nuclei Image Segmentation2023
- A Medical Semantic-Assisted Transformer for Radiographic Report Generation2022
- Boundary-Assisted Region Proposal Networks for Nucleus Segmentation2020
- Improving Nuclei/Gland Instance Segmentation in Histopathology Images by Full Resolution Neural Network and Spatial Constrained Loss2019
- PanNuke: An Open Pan-Cancer Histology Dataset for Nuclei Instance Segmentation and Classification2019
- CIA-Net: Robust Nuclei Instance Segmentation with Contour-Aware Information Aggregation2019