PromptSmooth: Certifying Robustness of Medical Vision-Language Models via Prompt Learning
Preprint
Abstract
Medical vision-language models (Med-VLMs) trained on large datasets of medical image-text pairs and later fine-tuned for specific tasks have emerged as a mainstream paradigm in medical image analysis. However, recent studies have highlighted the susceptibility of these Med-VLMs to adversarial attacks, raising concerns about their safety and robustness. Randomized smoothing is a well-known technique for turning any classifier into a model that is certifiably robust to adversarial perturbations. However, this approach requires retraining the Med-VLM-based classifier so that it classifies well under Gaussian noise, which is often infeasible in practice. In this paper, we propose a novel framework called PromptSmooth to achieve efficient certified robustness of Med-VLMs by leveraging the concept of prompt learning. Given any pre-trained Med-VLM, PromptSmooth adapts it to handle Gaussian noise by learning textual prompts in a zero-shot or few-shot manner, achieving a delicate balance between accuracy and robustness, while minimizing the computational overhead. Moreover, PromptSmooth requires only a single model to handle multiple noise levels, which substantially reduces the computational cost compared to traditional methods that rely on training a separate model for each noise level. Comprehensive experiments based on three Med-VLMs and across six downstream datasets of various imaging modalities demonstrate the efficacy of PromptSmooth. Our code and models are available at https://github.com/nhussein/promptsmooth.Keywords
All Related Versions
This publication has 10 references indexed in Scilit:
- A Foundation Language-Image Model of the Retina (FLAIR): encoding expert knowledge in text supervisionMedical Image Analysis, 2024
- Prompt-MIL: Boosting Multi-instance Learning Schemes via Task-Specific Prompt Tuning2023
- A visual–language foundation model for pathology image analysis using medical TwitterNature Medicine, 2023
- Deep learning for the detection of anatomical tissue structures and neoplasms of the skin on scanned histopathological tissue sectionsFrontiers in Oncology, 2022
- Learning to Prompt for Vision-Language ModelsInternational Journal of Computer Vision, 2022
- MedCLIP: Contrastive Learning from Unpaired Medical Images and Text2022
- Going deeper through the Gleason scoring scale: An automatic end-to-end system for histology prostate grading and cribriform pattern detectionComputer Methods and Programs in Biomedicine, 2020
- Adversarial attacks on medical machine learningScience, 2019
- Predicting survival from colorectal cancer histology slides using deep learning: A retrospective multicenter studyPLOS Medicine, 2019
- Augmenting the National Institutes of Health Chest Radiograph Dataset with Expert Annotations of Possible PneumoniaRadiology: Artificial Intelligence, 2019