PRISM: A Promptable and Robust Interactive Segmentation Model with Visual Prompts
Authors:
- 3 October 2024
- book chapter
- Published by Springer Nature
- Vol. 15003, 389-399
Abstract
In this paper, we present PRISM, a Promptable and Robust Interactive Segmentation Model, aiming for precise segmentation of 3D medical images. PRISM accepts various visual inputs, including points, boxes, and scribbles as sparse prompts, as well as masks as dense prompts. Specifically, PRISM is designed with four principles to achieve robustness: (1) Iterative learning. The model produces segmentations by using visual prompts from previous iterations to achieve progressive improvement. (2) Confidence learning. PRISM employs multiple segmentation heads per input image, each generating a continuous map and a confidence score to optimize predictions. (3) Corrective learning. Following each segmentation iteration, PRISM employs a shallow corrective refinement network to reassign mislabeled voxels. (4) Hybrid design. PRISM integrates hybrid encoders to better capture both the local and global information. Comprehensive validation of PRISM is conducted using four public datasets for tumor segmentation in the colon, pancreas, liver, and kidney, highlighting challenges caused by anatomical variations and ambiguous boundaries in accurate tumor identification. Compared to state-of-the-art methods, both with and without prompt engineering, PRISM significantly improves performance, achieving results that are close to human levels. The code is publicly available at https://github.com/MedICL-VU/PRISM.Keywords
This publication has 30 references indexed in Scilit:
- 3DSAM-adapter: Holistic adaptation of SAM from 2D to 3D for promptable tumor segmentationMedical Image Analysis, 2024
- MA-SAM: Modality-agnostic SAM adaptation for 3D medical image segmentationMedical Image Analysis, 2024
- Medical Image Segmentation Using Deep Learning2023
- SAM-U: Multi-box Prompts Triggered Uncertainty Estimation for Reliable SAM in Medical Image2023
- The Liver Tumor Segmentation Benchmark (LiTS)Medical Image Analysis, 2022
- Predictive uncertainty estimation for out-of-distribution detection in digital pathologyMedical Image Analysis, 2022
- The Medical Segmentation DecathlonNature Communications, 2022
- MIDeepSeg: Minimally interactive segmentation of unseen objects from medical images using deep learningMedical Image Analysis, 2021
- The state of the art in kidney and kidney tumor segmentation in contrast-enhanced CT imaging: Results of the KiTS19 challengeMedical Image Analysis, 2020
- Gradient Boosted Trees for Corrective Learning2017