Performance of generative pre-trained transformers (GPTs) in Certification Examination of the College of Family Physicians of Canada
Open Access
- 28 May 2024
- journal article
- research article
- Published by BMJ in Family Medicine and Community Health
- Vol. 12 (Suppl 1) , e002626
Abstract
Introduction The application of large language models such as generative pre-trained transformers (GPTs) has been promising in medical education, and its performance has been tested for different medical exams. This study aims to assess the performance of GPTs in responding to a set of sample questions of short-answer management problems (SAMPs) from the certification exam of the College of Family Physicians of Canada (CFPC). Method Between August 8th and 25th, 2023, we used GPT-3.5 and GPT-4 in five rounds to answer a sample of 77 SAMPs questions from the CFPC website. Two independent certified family physician reviewers scored AI-generated responses twice: first, according to the CFPC answer key (ie, CFPC score), and second, based on their knowledge and other references (ie, Reviews’ score). An ordinal logistic generalised estimating equations (GEE) model was applied to analyse repeated measures across the five rounds. Result According to the CFPC answer key, 607 (73.6%) lines of answers by GPT-3.5 and 691 (81%) by GPT-4 were deemed accurate. Reviewer’s scoring suggested that about 84% of the lines of answers provided by GPT-3.5 and 93% of GPT-4 were correct. The GEE analysis confirmed that over five rounds, the likelihood of achieving a higher CFPC Score Percentage for GPT-4 was 2.31 times more than GPT-3.5 (OR: 2.31; 95% CI: 1.53 to 3.47; pConclusion In our study, we used GPT-3.5 and GPT-4 to answer complex, open-ended sample questions of the CFPC exam and showed that more than 70% of the answers were accurate, and GPT-4 outperformed GPT-3.5 in responding to the questions. Large language models such as GPTs seem promising for assisting candidates of the CFPC exam by providing potential answers. However, their use for family medicine education and exam preparation needs further studies.Keywords
This publication has 20 references indexed in Scilit:
- ChatGPT in healthcare: A taxonomy and systematic reviewComputer Methods and Programs in Biomedicine, 2024
- Application of ChatGPT in multilingual medical education: How does ChatGPT fare in 2023's Iranian residency entrance examinationInformatics in Medicine Unlocked, 2023
- Evaluating the limits of AI in medical specialisation: ChatGPT’s performance on the UK Neurology Specialty Certificate ExaminationBMJ Neurology Open, 2023
- Performance of ChatGPT on a Radiology Board-style Examination: Insights into Current Strengths and LimitationsRadiology, 2023
- The next paradigm shift? ChatGPT, artificial intelligence, and medical educationMedical Teacher, 2023
- ChatGPT Performs on the Chinese National Medical Licensing ExaminationResearch Square, 2023
- Performance of ChatGPT on USMLE: Potential for AI-assisted medical education using large language modelsPLOS Digital Health, 2023
- ChatGPT for Clinical Vignette Generation, Revision, and EvaluationmedRxiv, 2023
- ChatGPT - Reshaping medical education and clinical managementPakistan Journal of Medical Sciences, 2023
- Application of Artificial Intelligence in Community-Based Primary Health Care: Systematic Scoping Review and Critical AppraisalJournal of Medical Internet Research, 2021