BAPLe: Backdoor Attacks on Medical Foundational Models using Prompt Learning
Preprint
Abstract
Medical foundation models are gaining prominence in the medical community for their ability to derive general representations from extensive collections of medical image-text pairs. Recent research indicates that these models are susceptible to backdoor attacks, which allow them to classify clean images accurately but fail when specific triggers are introduced. However, traditional backdoor attacks necessitate a considerable amount of additional data to maliciously pre-train a model. This requirement is often impractical in medical imaging applications due to the usual scarcity of data. Inspired by the latest developments in learnable prompts, this work introduces a method to embed a backdoor into the medical foundation model during the prompt learning phase. By incorporating learnable prompts within the text encoder and introducing imperceptible learnable noise trigger to the input images, we exploit the full capabilities of the medical foundation models (Med-FM). Our method, BAPLe, requires only a minimal subset of data to adjust the noise trigger and the text prompts for downstream tasks, enabling the creation of an effective backdoor attack. Through extensive experiments with four medical foundation models, each pre-trained on different modalities and evaluated across six downstream datasets, we demonstrate the efficacy of our approach. BAPLe achieves a high backdoor success rate across all models and datasets, outperforming the baseline backdoor attack methods. Our work highlights the vulnerability of Med-FMs towards backdoor attacks and strives to promote the safe adoption of Med-FMs before their deployment in real-world applications. Code is available at https://asif-hanif.github.io/baple/.Keywords
All Related Versions
This publication has 12 references indexed in Scilit:
- Towards generalist foundation model for radiology by leveraging web-scale 2D&3D medical dataNature Communications, 2025
- Foundation Models Defining a New Era in Vision: A Survey and OutlookIEEE Transactions on Pattern Analysis and Machine Intelligence, 2025
- A Foundation Language-Image Model of the Retina (FLAIR): encoding expert knowledge in text supervisionMedical Image Analysis, 2024
- Prompt-MIL: Boosting Multi-instance Learning Schemes via Task-Specific Prompt Tuning2023
- Backdoor attack and defense in federated generative adversarial network-based medical image synthesisMedical Image Analysis, 2023
- A visual–language foundation model for pathology image analysis using medical TwitterNature Medicine, 2023
- Learning to Prompt for Vision-Language ModelsInternational Journal of Computer Vision, 2022
- Backdoor Learning: A SurveyIEEE Transactions on Neural Networks and Learning Systems, 2022
- Exploring the effect of image enhancement techniques on COVID-19 detection using chest X-ray imagesComputers in Biology and Medicine, 2021
- Predicting survival from colorectal cancer histology slides using deep learning: A retrospective multicenter studyPLOS Medicine, 2019