Comparing Scoring Consistency of Large Language Models with Faculty for Formative Assessments in Medical Education
- 14 October 2024
- journal article
- research article
- Published by Springer Nature in Journal of General Internal Medicine
- Vol. 40 (1) , 127-134
Abstract
Background The Liaison Committee on Medical Education requires that medical students receive individualized feedback on their self-directed learning skills. Pre-clinical students are asked to complete multiple spaced critical appraisal assignments. However, the individual feedback requires significant faculty time. As large language models (LLMs) can score and generate feedback, we explored their use in grading formative assessments through validity and feasibility lenses. Objective To explore the consistency and feasibility of using an LLM to assess and provide feedback for formative assessments in undergraduate medical education. Design and Participants This was a cross-sectional study of pre-clinical students’ critical appraisal assignments at University of Illinois College of Medicine (UICOM) during the 2022–2023 academic year. Intervention An initial sample of ten assignments was used to develop a prompt. For each student entry, the de-identified assignment and prompt were provided to ChatGPT 3.5, and its scoring was compared to the existing faculty grade. Main Measures Differences in scoring of individual items between ChatGPT and faculty were assessed. Scoring consistency using inter-rater reliability (IRR) was calculated as percent exact agreement. Chi-squared test was used to determine if there were significant differences in scores. Psychometric characteristics including internal-consistency reliability, area under precision-recall curve (AUCPR), and cost were studied. Key Results In this cross-sectional study, 111 pre-clinical students’ faculty graded assignments were compared with that of ChatGPT and the scoring of individual items was comparable. The overall agreement between ChatGPT and faculty was 67% (OR = 2.53, P < 0.001); mean AUCPR was 0.69 (range 0.61–0.76). Internal-consistency reliability of ChatGPT was 0.64 and its use resulted in a fivefold reduction in faculty time, and potential savings of 150 faculty hours. Conclusions This study of psychometric characteristics of ChatGPT demonstrates the potential role for LLMs to assist faculty in assessing and providing feedback for formative assignments.Keywords
This publication has 30 references indexed in Scilit:
- Diabetes and artificial intelligence beyond the closed loop: a review of the landscape, promise and challengesDiabetologia, 2023
- A step-by-step researcher's guide to the use of an AI-based transformer in epidemiology: an exploratory analysis of ChatGPT using the STROBE checklist for observational studiesJournal of Public Health, 2023
- ChatGPT in Medical Education: a Paradigm Shift or a Dangerous Tool?Academic Psychiatry, 2023
- The rise of ChatGPT: Exploring its potential in medical educationAnatomical Sciences Education, 2023
- ChatGPT for good? On opportunities and challenges of large language models for educationLearning and Individual Differences, 2023
- Automatic essay exam scoring system: a systematic literature reviewProcedia Computer Science, 2023
- An automated essay scoring systems: a systematic literature reviewArtificial Intelligence Review, 2021
- Inter-rater reliability and generalizability of patient note scores using a scoring rubric based on the USMLE Step-2 CS formatAdvances in Health Sciences Education, 2016
- Using natural language processing to provide personalized learning opportunities from trainee clinical notesJournal of Biomedical Informatics, 2015
- A hierarchical classification approach to automated essay scoringAssessing Writing, 2015