RadCLIP: Enhancing Radiologic Image Analysis through Contrastive Language-Image Pre-training
Authors:
Preprint
Abstract
The integration of artificial intelligence (AI) with radiology has marked a transformative era in medical diagnostics. Vision foundation models have been adopted to enhance radiologic imaging analysis. However, the distinct complexities of radiological imaging, including the interpretation of 2D and 3D radiological data, pose unique challenges that existing models, trained on general non-medical images, fail to address adequately. To bridge this gap and capitalize on the diagnostic precision required in medical imaging, we introduce RadCLIP: a pioneering cross-modal foundational model that harnesses Contrastive Language-Image Pre-training (CLIP) to refine radiologic image analysis. RadCLIP incorporates a novel 3D slice pooling mechanism tailored for volumetric image analysis and is trained using a comprehensive and diverse dataset of radiologic image-text pairs. Our evaluations demonstrate that RadCLIP effectively aligns radiological images with their corresponding textual annotations, and in the meantime, offers a robust vision backbone for radiologic imagery with significant promise.Keywords
All Related Versions
This publication has 21 references indexed in Scilit:
- PMC-CLIP: Contrastive Language-Image Pre-training Using Biomedical Documents2023
- CLIP-Lung: Textual Knowledge-Guided Lung Nodule Malignancy Prediction2023
- CXR-CLIP: Toward Large Scale Chest X-ray Language-Image Pre-training2023
- M-FLAG: Medical Vision-Language Pre-training with Frozen Language Models and Latent Space Geometry Optimization2023
- Fine-tuned CLIP Models are Efficient Video Learners2023
- MedMNIST v2 - A large-scale lightweight benchmark for 2D and 3D biomedical image classificationScientific Data, 2023
- Swin Transformer: Hierarchical Vision Transformer using Shifted Windows2021
- GLoRIA: A Multimodal Global-Local Representation Learning Framework for Label-efficient Medical Image Recognition2021
- Rethinking Spatiotemporal Feature Learning: Speed-Accuracy Trade-offs in Video Classification2018
- Deep Residual Learning for Image Recognition2016