SAMCT: Segment Any CT Allowing Labor-Free Task-Indicator Prompts
- 7 November 2024
- journal article
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Transactions on Medical Imaging
- Vol. 44 (3) , 1386-1399
Abstract
Segment anything model (SAM), a foundation model with superior versatility and generalization across diverse segmentation tasks, has attracted widespread attention in medical imaging. However, it has been proved that SAM would encounter severe performance degradation due to the lack of medical knowledge in training and local feature encoding. Though several SAM-based models have been proposed for tuning SAM in medical imaging, they still suffer from insufficient feature extraction and highly rely on high-quality prompts. In this paper, we propose a powerful foundation model SAMCT allowing labor-free prompts and train it on a collected large CT dataset consisting of 1.1M CT images and 5M masks from public datasets. Specifically, based on SAM, SAMCT is further equipped with a U-shaped CNN image encoder, a cross-branch interaction module, and a task-indicator prompt encoder. The U-shaped CNN image encoder works in parallel with the ViT image encoder in SAM to supplement local features. Cross-branch interaction enhances the feature expression capability of the CNN image encoder and the ViT image encoder by exchanging global perception and local features from one to the other. The task-indicator prompt encoder is a plug-and-play component to effortlessly encode task-related indicators into prompt embeddings. In this way, SAMCT can work in an automatic manner in addition to the semi-automatic interactive strategy in SAM. Extensive experiments demonstrate the superiority of SAMCT against the state-of-the-art task-specific and SAM-based medical foundation models on various tasks. The code, data, and model checkpoints are available at https://github.com/xianlin7/SAMCT.Keywords
All Related Versions
Funding Information
- Fundamental Research Funds for the Central Universities (HUST:2024JYCXJJ032)
- National Natural Science Foundation of China (62202179, 62271220)
- Research Fund of Shenzhen (2024534319)
This publication has 46 references indexed in Scilit:
- Beyond Adapting SAM: Towards End-to-End Ultrasound Image Segmentation via Auto Prompting2024
- MedNeXt: Transformer-Driven Scaling of ConvNets for Medical Image Segmentation2023
- HaN‐Seg: The head and neck organ‐at‐risk CT and MR segmentation datasetMedical Physics, 2023
- Swin UNETR: Swin Transformers for Semantic Segmentation of Brain Tumors in MRI Images2022
- Pediatric chest‐abdomen‐pelvis and abdomen‐pelvis CT images with expert organ contoursMedical Physics, 2022
- TransFuse: Fusing Transformers and CNNs for Medical Image Segmentation2021
- Toward data‐efficient learning: A benchmark for COVID‐19 CT lung and infection segmentationMedical Physics, 2020
- Improving Uncertainty Estimation in Convolutional Neural Networks Using Inter-rater Agreement2019
- Evaluation of segmentation methods on head and neck CT: Auto‐segmentation challenge 2015Medical Physics, 2017
- U-Net: Convolutional Networks for Biomedical Image Segmentation2015