Prompt injection attacks on vision language models in oncology
Top Cited Papers
Open Access
- 1 February 2025
- journal article
- research article
- Published by Springer Nature in Nature Communications
- Vol. 16 (1) , 1-9
Abstract
Vision-language artificial intelligence models (VLMs) possess medical knowledge and can be employed in healthcare in numerous ways, including as image interpreters, virtual scribes, and general decision support systems. However, here, we demonstrate that current VLMs applied to medical tasks exhibit a fundamental security flaw: they can be compromised by prompt injection attacks. These can be used to output harmful information just by interacting with the VLM, without any access to its parameters. We perform a quantitative study to evaluate the vulnerabilities to these attacks in four state of the art VLMs: Claude-3 Opus, Claude-3.5 Sonnet, Reka Core, and GPT-4o. Using a set of N = 594 attacks, we show that all of these models are susceptible. Specifically, we show that embedding sub-visual prompts in manifold medical imaging data can cause the model to provide harmful output, and that these prompts are non-obvious to human observers. Thus, our study demonstrates a key vulnerability in medical VLMs which should be mitigated before widespread clinical adoption.Keywords
All Related Versions
This publication has 20 references indexed in Scilit:
- A multimodal generative AI copilot for human pathologyNature, 2024
- Comparative Analysis of Multimodal Large Language Model Performance on Clinical Vignette QuestionsJAMA, 2024
- Adapted large language models can outperform medical experts in clinical text summarizationNature Medicine, 2024
- Large language models in medicineNature Medicine, 2023
- Large language models encode clinical knowledgeNature, 2023
- The elephant in the room: cybersecurity in healthcareJournal of Clinical Monitoring and Computing, 2023
- Adversarial attacks and adversarial robustness in computational pathologyNature Communications, 2022
- DOMtegrity: ensuring web page integrity against malicious browser extensionsInternational Journal of Information Security, 2019
- Assessment of Employee Susceptibility to Phishing Attacks at US Health Care InstitutionsJAMA Network Open, 2019
- Wild patterns: Ten years after the rise of adversarial machine learningPattern Recognition, 2018