MedROV: Towards Real-Time Open-Vocabulary Detection Across Diverse Medical Imaging Modalities
Preprint
Abstract
Traditional object detection models in medical imaging operate within a closed-set paradigm, limiting their ability to detect objects of novel labels. Open-vocabulary object detection (OVOD) addresses this limitation but remains underexplored in medical imaging due to dataset scarcity and weak text-image alignment. To bridge this gap, we introduce MedROV, the first Real-time Open Vocabulary detection model for medical imaging. To enable open-vocabulary learning, we curate a large-scale dataset, Omnis, with 600K detection samples across nine imaging modalities and introduce a pseudo-labeling strategy to handle missing annotations from multi-source datasets. Additionally, we enhance generalization by incorporating knowledge from a large pre-trained foundation model. By leveraging contrastive learning and cross-modal representations, MedROV effectively detects both known and novel structures. Experimental results demonstrate that MedROV outperforms the previous state-of-the-art foundation model for medical image detection with an average absolute improvement of 40 mAP50, and surpasses closed-set detectors by more than 3 mAP50, while running at 70 FPS, setting a new benchmark in medical detection. Our source code, dataset, and trained model are available at https://github.com/toobatehreem/MedROV.Keywords
All Related Versions
This publication has 28 references indexed in Scilit:
- Grounding DINO: Marrying DINO with Grounded Pre-training for Open-Set Object Detection2024
- BGF-YOLO: Enhanced YOLOv8 with Multiscale Attentional Feature Fusion for Brain Tumor Detection2024
- SOCR‐YOLO: Small Objects Detection Algorithm in Medical ImagesInternational Journal of Imaging Systems and Technology, 2024
- BHSD: A 3D Multi-class Brain Hemorrhage Segmentation Dataset2023
- LiverHccSeg: A publicly available multiphasic MRI dataset with liver and HCC tumor segmentations and inter-rater agreement analysisData in Brief, 2023
- Detecting Twenty-Thousand Classes Using Image-Level Supervision2022
- Toward data‐efficient learning: A benchmark for COVID‐19 CT lung and infection segmentationMedical Physics, 2020
- IDRiD: Diabetic Retinopathy – Segmentation and Grading ChallengeMedical Image Analysis, 2020
- Hover-Net: Simultaneous segmentation and classification of nuclei in multi-tissue histology imagesMedical Image Analysis, 2019
- WM-DOVA maps for accurate polyp highlighting in colonoscopy: Validation vs. saliency maps from physiciansComputerized Medical Imaging and Graphics, 2015