Evaluating the Performance of DeepSeek 3, Claude Sonnet 4, and Gemini 2.5 in the Chilean Medical Licensing Examination: Observational Study.
Authors:
- Anaís Aracelly Lancellotti GuajardoAGAnaís Aracelly Lancellotti Guajardo
- Oscar Jerez YañezOYOscar Jerez Yañez
- Vicente Alberto Edgardo Jesus Silva ArroyoVicente Alberto Edgardo Jesus Silva Arroyo
- Marcos Jeremías Giovanny Vera CartesMCMarcos Jeremías Giovanny Vera Cartes
- Álvaro Andrés Herrera AlcaínoÁAÁlvaro Andrés Herrera Alcaíno
Open Access
- 23 September 2025
- journal article
- Published by Servicio de Publicaciones de la Universidad de Murcia in Revista Española de Educación Médica
- Vol. 6 (5)
Abstract
Introduction: Artificial intelligences and their continuous improvement have revolutionized medical education, but their performance in specific evaluative contexts still requires further exploration. Methods: This study qualitatively evaluated and compared the performance of three state-of-the-art language models — Claude Sonnet 4, Gemini 2.5, and DeepSeek 3 — in simulations of the National Medical Knowledge Examination (EUNACOM) in Chile. Three mock exams with 180 questions each were used, covering various medical areas and question types, including those based on clinical cases. Results: The results show that all AI models consistently passed the exams, with Claude Sonnet 4 achieving the highest overall performance (89% accuracy) and the greatest consistency across attempts. Clinical case-based questions were answered more accurately than theoretical knowledge questions, highlighting the models' strength in contextual clinical reasoning. Claude excelled in Internal Medicine and Psychiatry, DeepSeek in Surgery, and Gemini demonstrated balanced performance. However, specific gaps were identified in areas such as Public Health and clinical follow-up, suggesting the need for model-specific adjustments. Conclusion: The findings support the educational potential of these tools but also emphasize the importance of their ethical, supervised, and complementary use alongside traditional medical training. This study contributes to understanding the emerging role of artificial intelligence in professional assessments, as well as its limitations and opportunities within the Chilean medical context. Introducción: La inteligencias artificial y su mejora continua han revolucionado la educación médica, pero su desempeño en contextos evaluativos específicos aún requiere mayor exploración. Métodos: Este estudio evaluó y comparó cualitativamente el desempeño de tres modelos de lenguaje de última generación —Claude Sonnet 4, Gemini 2.5 y DeepSeek 3— en simulaciones del Examen Nacional de Conocimientos Médicos (EUNACOM) en Chile. Se utilizaron tres exámenes simulados con 180 preguntas cada uno, que abarcaban diversas áreas médicas y tipos de preguntas, incluidas las basadas en casos clínicos. Resultados: Los resultados muestran que todos los modelos de IA aprobaron los exámenes de forma consistente, y Claude Sonnet 4 logró el mayor desempeño general (89% de precisión) y la mayor consistencia en todos los intentos. Las preguntas basadas en casos clínicos se respondieron con mayor precisión que las preguntas de conocimiento teórico, lo que destaca la fortaleza de los modelos en el razonamiento clínico contextual. Claude sobresalió en Medicina Interna y Psiquiatría, DeepSeek en Cirugía y Gemini demostró un desempeño equilibrado. Sin embargo, se identificaron deficiencias específicas en áreas como la salud pública y el seguimiento clínico, lo que sugiere la necesidad de realizar ajustes específicos a cada modelo. Conclusión: Los hallazgos respaldan el potencial educativo de estas herramientas, pero también enfatizan la importancia de su uso ético, supervisado y complementario a la formación médica tradicional. Este estudio contribuye a comprender el papel emergente de la inteligencia artificial en las evaluaciones profesionales, así como sus limitaciones y oportunidades en el contexto médico chileno.Keywords
This publication has 7 references indexed in Scilit:
- Artificial Intelligence in Health Professions Education assessment: AMEE Guide No. 178Medical Teacher, 2025
- Exploring the Performance of ChatGPT Versions 3.5, 4, and 4 With Vision in the Chilean Medical Licensing Examination: Observational StudyJMIR Medical Education, 2024
- Evaluating the Efficacy of ChatGPT in Navigating the Spanish Medical Residency Entrance Examination (MIR): Promising Horizons for AI in Clinical MedicineClinics and Practice, 2023
- The impact of Chat Generative Pre-trained Transformer (ChatGPT) on medical educationPostgraduate Medical Journal, 2023
- The Role of ChatGPT, Generative Language Models, and Artificial Intelligence in Medical Education: A Conversation With ChatGPT and a Call for PapersJMIR Medical Education, 2023
- Performance of ChatGPT on USMLE: Potential for AI-assisted medical education using large language modelsPLOS Digital Health, 2023
- ¿Es capaz “ChatGPT” de aprobar el examen MIR de 2022? Implicaciones de la inteligencia artificial en la educación médica en EspañaRevista Española de Educación Médica, 2022