Performance of DeepSeek-R1 and ChatGPT-4o on the Chinese National Medical Licensing Examination: A Comparative Study

Authors:
Top Cited Papers
Publisher website
Google Scholar
Abstract
Large Language Models (LLMs) have a significant impact on medical education due to their advanced natural language processing capabilities. ChatGPT-4o (Chat Generative Pre-trained Transformer), a mainstream Western LLM, demonstrates powerful multimodal abilities. DeepSeek-R1, a newly released free and open-source LLM from China, demonstrates capabilities on par with ChatGPT-4o across various domains. This study aims to evaluate the performance of DeepSeek-R1 and ChatGPT-4o on the Chinese National Medical Licensing Examination (CNMLE) and explore the performance differences of LLMs from distinct linguistic environments in Chinese medical education. We evaluated both LLMs using 600 multiple-choice questions from the written part of 2024 CNMLE, covering four units. The questions were categorized into low- and high-difficulty groups according to difficulty. The primary outcome was the overall accuracy rate of each LLM. The secondary outcomes included accuracy within each of the four units and within the two difficulty-level groups. DeepSeek-R1 achieved a statistically significantly higher overall accuracy of 92.0% compared to ChatGPT-4o’s 87.2% (P < 0.05). In the low-difficulty group, DeepSeek-R1 demonstrated an accuracy rate of 95.9%, which was significantly higher than ChatGPT-4o’s 92.0% (P < 0.05). No statistically significant differences were observed between the models in any of the four units or in the high-difficulty group (P > 0.05). DeepSeek-R1 demonstrated a performance advantage on CNMLE.
Funding Information
  • Zhenjiang Science and Technology Planning Project (No. SH2024017)

This publication has 23 references indexed in Scilit: