Is ChatGPT ‘ready’ to be a learning tool for medical undergraduates and will it perform equally in different subjects? Comparative study of ChatGPT performance in tutorial and case-based learning questions in physiology and biochemistry
- 31 January 2024
- journal article
- research article
- Published by Taylor & Francis in Medical Teacher
- Vol. 46 (11) , 1441-1447
Abstract
Generative AI will become an integral part of education in future. The potential of this technology in different disciplines should be identified to promote effective adoption. This study evaluated the performance of ChatGPT in tutorial and case-based learning questions in physiology and biochemistry for medical undergraduates. Our study mainly focused on the performance of GPT-3.5 version while a subgroup was comparatively assessed on GPT-3.5 and GPT-4 performances. Answers were generated in GPT-3.5 for 44 modified essay questions (MEQs) in physiology and 43 MEQs in biochemistry. Each answer was graded by two independent examiners. Subsequently, a subset of 15 questions from each subject were selected to represent different score categories of the GPT-3.5 answers; responses were generated in GPT-4, and graded. The mean score for physiology answers was 74.7 (SD 25.96). GPT-3.5 demonstrated a statistically significant (p = .009) superior performance in lower-order questions of Bloom's taxonomy in comparison to higher-order questions. Deficiencies in the application of physiological principles in clinical context were noted as a drawback. Scores in biochemistry were relatively lower with a mean score of 59.3 (SD 26.9) for GPT-3.5. There was no statistically significant difference in the scores for higher and lower-order questions of Bloom's taxonomy. The deficiencies highlighted were lack of in-depth explanations and precision. The subset of questions where the GPT-4 and GPT-3.5 were compared demonstrated a better overall performance in GPT-4 responses in both subjects. This difference between the GPT-3.5 and GPT-4 performance was statistically significant in biochemistry but not in physiology. The differences in performance across the two versions, GPT-3.5 and GPT-4 across the disciplines are noteworthy. Educators and students should understand the strengths and limitations of this technology in different fields to effectively integrate this technology into teaching and learning.Keywords
Funding Information
- Teaching Enhancement Grant (2022/2023)
- National University of Singapore (WBS E-560-00-0734-01)
This publication has 20 references indexed in Scilit:
- A SWOT analysis of ChatGPT: Implications for educational practice and researchInnovations in Education and Teaching International, 2023
- ChatGPT Utility in Healthcare Education, Research, and Practice: Systematic Review on the Promising Perspectives and Valid ConcernsHealthcare, 2023
- ChatGPT for good? On opportunities and challenges of large language models for educationLearning and Individual Differences, 2023
- Chatting and cheating: Ensuring academic integrity in the era of ChatGPTInnovations in Education and Teaching International, 2023
- ChatGPT applications in medical, dental, pharmacy, and public health education: A descriptive study highlighting the advantages and limitationsNarra J, 2023
- Performance of ChatGPT on USMLE: Potential for AI-assisted medical education using large language modelsPLOS Digital Health, 2023
- Academic Integrity considerations of AI Large Language Models in the post-pandemic era: ChatGPT and beyondJournal of University Teaching and Learning Practice, 2023
- ChatGPT Goes to Law SchoolSSRN Electronic Journal, 2023
- Daily briefing: Will ChatGPT kill the essay assignment?Nature, 2022
- Do Large Language Models Understand Us?Daedalus, 2022