Can a Generative Artificial Intelligence Model Be Used to Create Mass Casualty Incident Simulation Scenarios? A Feasibility Study
Authors:
- Sergio M. NavarroSergio M. Navarro
- Angie G. AtkinsonAAAngie G. Atkinson
- Ege DonagayEDEge Donagay
- Maxwell JabaayMJMaxwell Jabaay
- Sarah LundSLSarah Lund
- Myung S. ParkMPMyung S. Park
- Erica A. LoomisELErica A. Loomis
- John M. ZietlowJZJohn M. Zietlow
- T. N. Diem VuTVT. N. Diem Vu
- Mariela RiveraMRMariela Rivera
- Daniel StephensDSDaniel Stephens
Open Access
- 5 December 2025
- journal article
- research article
- Published by MDPI AG in Healthcare
- Vol. 13 (24) , 3184
Abstract
Introduction: Mass casualty incident (MCI) simulation scenarios are developed based on detailed review and planning by multidisciplinary trauma teams. This study aimed to assess the feasibility of using generative artificial intelligence (AI) in developing mass casualty trauma simulation scenarios. The study evaluated a range of mass casualty trauma simulation scenarios generated from a public generative artificial intelligence platform based on publicly available data with a validated objective simulation scoring tool. Methods: Using a large language model (LLM) platform (ChatGPT4, OpenAI, San Francisco, CA, USA), 10 complex MCI trauma simulation scenarios were generated based on publicly available US reported trauma data. Each scenario was evaluated by two Advanced Trauma Life Support (ATLS) certified raters based on the Simulation Scenario Evaluation Tool (SSET), a validated scoring tool out of 100 points. The tool scoring is based on learning objectives, tasks for performance, clinical progression, debriefing criteria, and resources. Two publicly available mass casualty trauma scenarios were similarly evaluated as controls. Revision and recommended feedback was provided for the scenarios, with review time recorded. Post-revision scenarios were evaluated. Interrater reliability was calculated based on Intraclass Correlation Coefficients (2, k) (ICCs). For the scenarios, scores and review times were reported as medians with interquartile range (IQR) as 25th and 75th percentiles. Results: Ten mass casualty trauma simulation scenarios were generated by an LLM, producing a total of 62 simulated patients. The initial LLM-generated scenarios demonstrated a median SSET score of 78.5 (IQR 74–82), substantially lower than the median score of 94 (IQR 93–95) observed in publicly available scenarios. The interrater reliability ICC for the LLM-generated scenarios was 0.965 and 1.00 for publicly available scenarios. Following secondary human revision and iterative refinement, the LLM-generated scenarios improved, achieving a median SSET score of 94 (IQR 93–96) with an interrater reliability ICC of 0.7425. Conclusions: The feasibility study suggests that a structured, collaborative workflow combining LLM-based generation with expert human review may enable a new approach to mass casualty trauma simulation scenario creation. LLMs hold promise as a scalable tool for the development of MCI training materials. However, consistent human oversight, quality assurance processes, and governance frameworks remain essential to ensure clinical accuracy, safety, and educational value.Keywords
Funding Information
- Stimulating Access to Research in Residency (R38HL150086)
- National Center for Advancing Translational Sciences (UL1 TR002377)
This publication has 30 references indexed in Scilit:
- AI-Generated Clinical Summaries Require More Than AccuracyJAMA, 2024
- Affiliation Bias in Peer Review of Abstracts by a Large Language ModelJAMA, 2024
- Review of the requirements for effective mass casualty preparedness for trauma systems. A disaster waiting to happen?British Journal of Anaesthesia, 2022
- Exercising for mass casualty preparednessBritish Journal of Anaesthesia, 2022
- Refining mass casualty plans with simulation-based iterative learningBritish Journal of Anaesthesia, 2022
- Mass Casualty Incident (MCI) training in a metropolitan university hospital: short-term experience with MAss Casualty SIMulation system MACSIM®European Journal of Trauma and Emergency Surgery, 2020
- Evaluation of Combined Artificial Intelligence and Radiologist Assessment to Interpret Screening MammogramsJAMA Network Open, 2020
- A systematic literature review of criteria and models for casualty distribution in trauma related mass casualty incidentsInjury, 2018
- SIMEDIS: a Discrete-Event Simulation Model for Testing Responses to Mass Casualty IncidentsJournal of Medical Systems, 2016
- A Guideline of Selecting and Reporting Intraclass Correlation Coefficients for Reliability ResearchJournal of Chiropractic Medicine, 2016