ChatGPT versus expert feedback on clinical reasoning questions and their effect on learning: a randomized controlled trial
Top Cited Papers
Open Access
- 5 December 2024
- journal article
- research article
- Published by Oxford University Press (OUP) in Postgraduate Medical Journal
- Vol. 101 (1195) , 458-463
Abstract
Purpose: To evaluate the effectiveness of ChatGPT-generated feedback compared to expert-written feedback in improving clinical reasoning skills among first-year medical students. Methods: This is a randomized controlled trial conducted at a single medical school and involved 129 first-year medical students who were randomly assigned to two groups. Both groups completed three formative tests with feedback on urinary tract infections (UTIs; uncomplicated, complicated, pyelonephritis) over five consecutive days as a spaced repetition, receiving either expert-written feedback (control, n = 65) or ChatGPT-generated feedback (experiment, n = 64). Clinical reasoning skills were assessed using Key-Features Questions (KFQs) immediately after the intervention and 10 days later. Students’ critical approach to artificial intelligence (AI) was also measured before and after disclosing the AI involvement in feedback generation. Results: There was no significant difference between the mean scores of the control (immediate: 78.5 ± 20.6 delayed: 78.0 ± 21.2) and experiment (immediate: 74.7 ± 15.1, delayed: 76.0 ± 14.5) groups in overall performance on Key-Features Questions (out of 120 points) immediately (P = .26) or after 10 days (P = .57), with small effect sizes. However, the control group outperformed the ChatGPT group in complicated urinary tract infection cases (P < .001). The experiment group showed a significantly higher critical approach to AI after disclosing, with medium-large effect sizes. Conclusions: ChatGPT-generated feedback can be an effective alternative to expert feedback in improving clinical reasoning skills in medical students, particularly in resource-constrained settings with limited expert availability. However, AI-generated feedback may lack the nuance needed for more complex cases, emphasizing the need for expert review. Additionally, exposure to the drawbacks in AI-generated feedback can enhance students’ critical approach towards AI-generated educational content.Keywords
Funding Information
- Scientific and Technological Research Council of Türkiye
This publication has 26 references indexed in Scilit:
- Leveraging large language models to construct feedback from medical multiple-choice QuestionsScientific Reports, 2024
- Can ChatGPT generate practice question explanations for medical students, a new faculty teaching tool?Medical Teacher, 2024
- ChatGPT to generate clinical vignettes for teaching and multiple-choice questions for assessment: A randomized controlled experimentMedical Teacher, 2024
- A scoping review of artificial intelligence in medical education: BEME Guide No. 84Medical Teacher, 2024
- Exploratory Factor Analysis of a Computerized Case-Based F-Type Testlet VariantMedical Science Educator, 2023
- Medical Teacher’s first ChatGPT’s referencing hallucinations: Lessons for editors, reviewers, and teachersMedical Teacher, 2023
- The need for longitudinal clinical reasoning teaching and assessment: Results of an international surveyMedical Teacher, 2020
- Productive failure as an instructional approach to promote future learningAdvances in Health Sciences Education, 2019
- The key-features approach to assess clinical decisions: validity evidence to dateAdvances in Health Sciences Education, 2018
- Social Workers as Patient Navigators for Breast Cancer Survivors: What Do African-American Medically Underserved Women Think of This Idea?Social Work in Health Care, 2009