Use of Large Language Models for Automated Short-Answer Question Grading in Second-Cycle Medical Education: Performance and Consistency Evaluation Study
Open Access
- 2 November 2026
- journal article
- Published by JMIR Publications Inc. in JMIR Medical Education
- Vol. 12, e89445
Abstract
Background: Short-answer questions (SAQs) effectively assess higher-order reasoning skills in medical education but face challenges in scalability due to time-intensive manual grading and concerns about interrater reliability. Recent advances in large language models (LLMs) may offer a solution through automated assessment. Objective: This study evaluated the performance and consistency of LLMs—ChatGPT, Claude Sonnet 4, and Mistral Pro—compared with those of human graders for SAQ grading in hepatology and gastroenterology. Methods: We retrospectively analyzed SAQs from hepatology and gastroenterology examinations at Sorbonne University, Paris, France (2023-2025). Three human evaluators (2 hepatogastroenterologists and 1 radiologist) independently graded all responses twice, with the majority consensus score serving as the reference standard. Six LLMs were evaluated: ChatGPT (GPT-4o, GPT-4.5, GPT-4o3, and GPT-5), Claude Sonnet 4, and Mistral Pro. Each LLM graded all responses twice using standardized prompts, including questions, model answers, grading guidelines, and scoring rubrics. Agreement was assessed using quadratic weighted kappa (QWK), Cohen κ, the intraclass correlation coefficient (ICC), and the mean absolute error (MAE) for interhuman, intrahuman, LLM-versus-human, and intramodel comparisons. Results: A total of 902 student responses from 12 SAQs were included and analyzed. Interhuman grader agreement was almost perfect (QWK 0.90-0.92; ICC 0.90-0.92; MAE 0.05-0.07). Claude Sonnet 4 and GPT-4o3 demonstrated near-human–level performance when compared with the majority vote of human evaluators (Claude Sonnet 4: QWK 0.88-0.90; ICC 0.88-0.90; MAE 0.06-0.07 and GPT-4o3: QWK 0.82-0.89; ICC 0.82-0.89; MAE 0.07-0.09). GPT-5 showed substantial agreement (QWK 0.77-0.79). Mistral Pro and older ChatGPT versions performed poorly (QWK <0.63). Intramodel consistency was the highest for Claude Sonnet 4 (QWK 0.93), approaching the best human intrarater reliability. Agreement did not correlate with prompt design choices (number of examples or minimum word count). Conclusions: Advanced LLMs, particularly Claude Sonnet 4 and GPT-4o3, demonstrated near-human–level grading performance, supporting their potential use as assistive tools in SAQ grading. However, performance varied by model, and agreement alone does not ensure educational validity. Until broader multicenter validation is conducted across diverse domains, stakes, and question formats—along with safeguards for bias, privacy, and model drift—LLMs should augment rather than replace human faculty graders.Keywords
All Related Versions
This publication has 21 references indexed in Scilit:
- Evaluating large language models as graders of medical short answer questions: a comparative analysis with expert human gradersMedical Education Online, 2025
- Can AI grade like a professor? comparing artificial intelligence and faculty scoring of medical student short-answer clinical reasoning examsAdvances in Health Sciences Education, 2025
- Using Large Language Models for Automated Grading of Student Writing about ScienceInternational Journal of Artificial Intelligence in Education, 2025
- Can Large Language Models Make the Grade? An Empirical Study Evaluating LLMs Ability To Mark Short Answer Questions in K-12 Education2024
- Prompt engineering in consistency and reliability with the evidence-based guideline for LLMsnpj Digital Medicine, 2024
- An experimental comparison of multiple-choice and short-answer questions on a high-stakes test for medical studentsAdvances in Health Sciences Education, 2023
- Practicing retrieval in university teaching: short-answer questions are beneficial, whereas multiple-choice questions are notJournal of Cognitive Psychology, 2022
- Very‐short‐answer questions: reliability, discrimination and acceptabilityMedical Education, 2018
- The Eras and Trends of Automatic Short Answer GradingInternational Journal of Artificial Intelligence in Education, 2015
- Validity of final examinations in undergraduate medical trainingBMJ, 2000