Comparison of ChatGPT-4o and expert evaluation in endodontic education: a cross-sectional pilot study
Open Access
- 1 December 2025
- journal article
- research article
- Published by Springer Nature in BMC Medical Education
- Vol. 26 (1)
Abstract
Background: Artificial intelligence (AI) has the potential to enhance objectivity and scalability in educational assessment, yet its role in evaluating technical dental skills remains unclear. This study aimed to compare ChatGPT-4o based assessments with expert evaluations in undergraduate endodontic training and to explore student perceptions of AI-assisted feedback. Methods: This cross-sectional pilot study was conducted during the 2024–2025 academic year with 32 dental students from a faculty of dentistry, who completed root canal treatments. Postoperative radiographs were evaluated by 10 years experienced endodontist and ChatGPT-4o was used to evaluate performance based on five standardized criteria: canal centering, homogeneity, procedural errors, apical shaping, and overall taper, each rated on a 5-point Likert scale. Inter-rater reliability was assessed via intraclass correlation coefficients (ICC), and Pearson correlation tested linear alignment. Students rated both feedback sources via Likert-scale questionnaires and open-ended comments; paired-sample t-tests compared the mean scores. Results: Agreement between AI and expert evaluation was limited, with ICC ranging from 0.36 to 0.45, indicating poor to moderate reliability. Pearson r values were < 0.3 and not statistically significant, demonstrating weak linear correlation. While students rated AI-generated feedback as moderately useful, expert feedback scored higher across educational value (mean 4.29 vs. 3.90), clinical reasoning support (4.19 vs. 4.06), and reliability (4.00 vs. 3.91); differences were not statistically significant. Notably, 53.1% of students preferred a combination of AI and expert feedback for optimal learning. Conclusions: AI-generated feedback was moderately useful to students, but expert feedback consistently scored higher. For complex psychomotor skills and radiographic interpretation, AI should serve as an auxiliary tool rather than an independent assessor. Further validation of advanced multimodal AI systems and development of hybrid frameworks combining algorithmic objectivity with expert judgment are recommended.Keywords
This publication has 26 references indexed in Scilit:
- ChatGPT versus expert feedback on clinical reasoning questions and their effect on learning: a randomized controlled trialPostgraduate Medical Journal, 2024
- Performance of ChatGPT‐3.5 and ChatGPT‐4o in the Japanese National Dental ExaminationJournal of Dental Education, 2024
- Assessing GPT-4 multimodal performance in radiological image analysisEuropean Radiology, 2024
- Evaluating the image recognition capabilities of GPT-4V and Gemini Pro in the Japanese national dental examinationJournal of Dental Sciences, 2024
- Unveiling the shadows: Beyond the hype of AI in educationHeliyon, 2024
- Enhancing systematic reviews in orthodontics: a comparative examination of GPT-3.5 and GPT-4 for generating PICO-based queries with tailored prompts and configurationsEuropean Journal of Orthodontics, 2024
- Evaluating GPT-4V’s performance in the Japanese national dental examination: A challenge exploredJournal of Dental Sciences, 2023
- The enlightening role of explainable artificial intelligence in medical & healthcare domains: A systematic literature reviewComputers in Biology and Medicine, 2023
- Effect of Artificial Intelligence Tutoring vs Expert Instruction on Learning Simulated Surgical Skills Among Medical StudentsJAMA Network Open, 2022
- Machine learning approaches in medical image analysis: From detection to diagnosisMedical Image Analysis, 2016