Leveraging simulation to provide a practical framework for estimating the novel scope of risk of large language models in healthcare
Open Access
- 24 June 2026
- journal article
- Published by BMJ in BMJ Mental Health
- Vol. 29 (1) , e302626
Abstract
Background: Large language models (LLMs) are rapidly entering clinical and consumer use, yet their probabilistic outputs have delivered a variety of unsafe user responses. Difficulties in quantifying and mitigating risks posed by LLMs threaten to stall regulatory evaluation and clinical deployment of LLM-based software as a medical device (LLM-SaMD). Practical approaches are needed to extend existing medical-device regulations to LLM-SaMDs. Objective: To demonstrate how simulation can extend existing medical-device risk management frameworks for addressing LLM-SaMD-specific risks. Methods: We implement a simulation-based methodology for estimating LLM-SaMD risk. Fourteen open-source models were evaluated on three safety-classification tasks: suicidal-ideation, therapy-request and therapy-like interaction detection. Synthetic datasets were generated by Gemini 2.5 Pro and evaluated by psychiatrists. Model false-negative rates informed estimates of P 1 , the likelihood that a hazard progresses to a hazardous situation, and P 2 , the likelihood that that situation results in harm. Findings: LLM success at generating synthetic datasets varied by task, with strong performance for neutral and non-therapeutic content but frequent errors in suicidal-ideation and therapy-like interactions. Performance generally improved with model size. Estimated P 1 values ranged from 1.1×10⁻⁸ to 1.6×10⁻⁴ and P 2 from 4.9×10⁻⁵ to 5.1×10⁻³, spanning four orders of magnitude. Conclusions: By linking model failure modes to structured pathways to harm, simulation can extend existing medical-device risk frameworks to help address the probabilistic and context-dependent risks of LLM-SaMDs. Clinical implications: Simulation-based risk estimation offers a practical way to characterise the risk landscape for specific LLM-SaMD, patient population and clinical context combinations.Keywords
This publication has 14 references indexed in Scilit:
- Harm Reduction Strategies for Thoughtful Use of Large Language Models in the Medical Domain: Perspectives for Patients and CliniciansJournal of Medical Internet Research, 2025
- Monitoring strategies for continuous evaluation of deployed clinical prediction modelsJournal of Biomedical Informatics, 2025
- Detecting and Remediating Harmful Data Shifts for the Responsible Deployment of Clinical AI ModelsJAMA Network Open, 2025
- A framework to assess clinical safety and hallucination rates of LLMs for medical text summarisationnpj Digital Medicine, 2025
- Medical large language models are vulnerable to data-poisoning attacksNature Medicine, 2025
- Tool Chain for the Virtual Validation of Autonomous VehiclesATZ worldwide, 2024
- Credibility assessment of patient-specific computational modeling using patient-specific cardiac modeling as an exemplarPLOS Computational Biology, 2022
- In silico trials: Verification, validation and uncertainty quantification of predictive models used in the regulatory evaluation of biomedical productsMethods, 2021
- From development to deployment: dataset shift, causality, and shift-stable models in health AIBiostatistics, 2019
- Advancing Regulatory Science With Computational Modeling for Medical Devices at the FDA's Office of Science and Engineering LaboratoriesFrontiers in Medicine, 2018