Evaluating large language models as graders of medical short answer questions: a comparative analysis with expert human graders
Top Cited Papers
Open Access
- 24 August 2025
- journal article
- research article
- Published by Taylor & Francis in Medical Education Online
- Vol. 30 (1) , 2550751
Abstract
The assessment of short-answer questions (SAQs) in medical education is resource-intensive, requiring significant expert time. Large Language Models (LLMs) offer potential for automating this process, but their efficacy in specialized medical education assessment remains understudied. To evaluate the capability of five LLMs to grade medical SAQs compared to expert human graders across four distinct medical disciplines. This study analyzed 804 student responses across anatomy, histology, embryology, and physiology. Three faculty members graded all responses. Five LLMs (GPT-4.1, Gemini, Claude, Copilot, DeepSeek) evaluated responses twice: first using their learned representations to generate their own grading criteria (A1), then using expert-provided rubrics (A2). Agreement was measured using Cohen's Kappa and Intraclass Correlation Coefficient (ICC). Expert-expert agreement was substantial across all questions (average Kappa: 0.69, ICC: 0.86), ranging from moderate (SAQ2: 0.57) to almost perfect (SAQ4: 0.87). LLM performance varied dramatically by question type and model. The highest expert-LLM agreement was observed for Claude on SAQ3 (Kappa: 0.61) and DeepSeek on SAQ2 (Kappa: 0.53). Providing expert criteria had inconsistent effects, significantly improving some model-question combinations while decreasing others. No single LLM consistently outperformed others across all domains. LLM strictness in grading unsatisfactory responses varied substantially from experts. LLMs demonstrated domain-specific variations in grading capabilities. The provision of expert criteria did not consistently improve performance. While LLMs show promise for supporting medical education assessment, their implementation requires domain-specific considerations and continued human oversight.Keywords
This publication has 32 references indexed in Scilit:
- Validity evidence supporting clinical skills assessment by artificial intelligence compared with trained clinician ratersMedical Education, 2023
- Automated grading of anatomical objective structured practical examinations using decision trees: An artificial intelligence approachAnatomical Sciences Education, 2023
- Prediction of medical sciences students’ performance on high-stakes examinations using machine learning models: a protocol for a systematic reviewBMJ Open, 2023
- Rethinking assessment in response to generative artificial intelligenceMedical Education, 2023
- Precision Medical EducationAcademic Medicine, 2023
- The fundamentals of Artificial Intelligence in medical education research: AMEE Guide No. 156Medical Teacher, 2023
- A Multi-institutional Study of the Feasibility and Reliability of the Implementation of Constructed Response Exam QuestionsTeaching and Learning in Medicine, 2022
- Twelve tips for introducing very short answer questions (VSAQs) into your medical curriculumMedical Teacher, 2022
- Should multiple-choice questions get the SAQ? Development of a short-answer question writing rubricCurrents in Pharmacy Teaching and Learning, 2022
- Revisiting Miller’s pyramid in medical education: the gap between traditional assessment and diagnostic reasoningInternational Journal of Medical Education, 2019