Towards medical AI misalignment: a preliminary study
Authors:
Preprint
Abstract
Despite their staggering capabilities as assistant tools, often exceeding human performances, Large Language Models (LLMs) are still prone to jailbreak attempts from malevolent users. Although red teaming practices have already identified and helped to address several such jailbreak techniques, one particular sturdy approach involving role-playing (which we named `Goofy Game') seems effective against most of the current LLMs safeguards. This can result in the provision of unsafe content, which, although not harmful per se, might lead to dangerous consequences if delivered in a setting such as the medical domain. In this preliminary and exploratory study, we provide an initial analysis of how, even without technical knowledge of the internal architecture and parameters of generative AI models, a malicious user could construct a role-playing prompt capable of coercing an LLM into producing incorrect (and potentially harmful) clinical suggestions. We aim to illustrate a specific vulnerability scenario, providing insights that can support future advancements in the field.Keywords
All Related Versions
This publication has 8 references indexed in Scilit:
- LLMs Will Always Hallucinate, and We Need to Live with This2025
- JailbreakRadar: Comprehensive Assessment of Jailbreak Attacks Against LLMs2025
- A Survey of Clinicians' Views of the Utility of Large Language ModelsApplied Clinical Informatics, 2024
- Language Models are Surprisingly Fragile to Drug Names in Biomedical Benchmarks2024
- Large language models encode clinical knowledgeNature, 2023
- EQRbot: A chatbot delivering EQR argument-based explanationsFrontiers in Artificial Intelligence, 2023
- Jailbroken: How Does LLM Safety Training Fail?2023
- Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena2023