Exploring the Use of a Large Language Model in Simulation Debriefing: An Observational Simulation-Based Pilot Study
Authors:
- Eury Ma HongEHEury Ma Hong
- Sundes KazmirSKSundes Kazmir
- Benjamin DylikBDBenjamin Dylik
- Marc Md AuerbachMAMarc Md Auerbach
- Matteo Bs RosatiMRMatteo Bs Rosati
- Sofia AthanasopoulouSASofia Athanasopoulou
- Russell HimmelsteinRHRussell Himmelstein
- Travis M. WhitfillTWTravis M. Whitfill
- Lindsay Md JohnstonLJLindsay Md Johnston
- Traci A. WolbrinkTWTraci A. Wolbrink
- Arielle Shibi Bs RosenARArielle Shibi Bs Rosen
- Isabel T. GrossIGIsabel T. Gross
- 6 May 2025
- journal article
- empirical investigations
- Published by Wolters Kluwer Health in Simulation in Healthcare: The Journal of the Society for Simulation in Healthcare
- Vol. 20 (6) , 366-371
Abstract
Introduction: Facilitating debriefings in simulation is a complex task with high task load. The increasing availability of generative artificial intelligence (AI) offers an opportunity to support facilitators. We explored simulation facilitation and debriefing strategies using a large language model (LLM) to decrease facilitators' task load and allow for a more comprehensive debrief. Methods: This prospective, observational, simulation-based pilot study was conducted at Yale University School of Medicine. For each simulation, a debriefing script was generated by passing a real-time transcription of the simulation case as input to the GPT-4o LLM. Thereafter, facilitators and learners completed surveys and task workload assessments. The primary outcome was the task workload as measured by the NASA-TLX scale. The secondary outcome was the perception of the AI technologies in the simulation, measured with survey-based questions. Results: This study involved four facilitators and 25 learners, with all data being self-reported. All showed strong enthusiasm for AI integration, with mean Likert scores of 4.75/5 and 4.0/5, respectively. NASA-TLX scores revealed moderate to high mental demand for facilitators (M = .8/21; SD = 6.4) and learners (M = 9.9/21; SD = 4.5). AI was perceived to help maintain focus (M = 4.8/5), support learning objectives (M = 4.2/5), and minimize distractions for both facilitators (M = 4.6/5) and teams (M = 4.5/5). Conclusions: This study highlights LLM integration in aiding debriefing by organizing complex information. Though facilitators reported a considerable task load, findings suggest that LLM can enhance simulation-based debrief quality, while there remains a continuous need for human oversight.Keywords
This publication has 10 references indexed in Scilit:
- Testing and Evaluation of Health Care Applications of Large Language ModelsJAMA, 2025
- A Language Model–Powered Simulated Patient With Automated Feedback for History Taking: Prospective StudyJMIR Medical Education, 2024
- A Generative Pretrained Transformer (GPT)–Powered Chatbot as a Simulated Patient to Practice History Taking: Prospective, Mixed Methods StudyJMIR Medical Education, 2024
- Debriefing Methods for Simulation in HealthcareSimulation in Healthcare: The Journal of the Society for Simulation in Healthcare, 2024
- Impact of the PEARLS Healthcare Debriefing cognitive aid on facilitator cognitive load, workload, and debriefing quality: a pilot studyAdvances in Simulation, 2022
- Artificial Intelligence Education for the Health Workforce: Expert Survey of Approaches and NeedsJMIR Medical Education, 2022
- Artificial Intelligence Education Programs for Health Care Professionals: Scoping ReviewJMIR Medical Education, 2021
- Siri, Siri, in my hand: Who’s the fairest in the land? On the interpretations, illustrations, and implications of artificial intelligenceBusiness Horizons, 2019
- The NASA Task Load Index as a measure of overall workload among neonatal, paediatric and adult intensive care nursesIntensive and Critical Care Nursing, 2018
- Promoting Excellence and Reflective Learning in Simulation (PEARLS)Simulation in Healthcare: The Journal of the Society for Simulation in Healthcare, 2015