Evaluation of ChatGPT as a diagnostic tool for medical learners and clinicians
Open Access
- 31 July 2024
- journal article
- research article
- Published by Public Library of Science (PLoS) in PLOS ONE
- Vol. 19 (7) , e0307383
Abstract
Background: ChatGPT is a large language model (LLM) trained on over 400 billion words from books, articles, and websites. Its extensive training draws from a large database of information, making it valuable as a diagnostic aid. Moreover, its capacity to comprehend and generate human language allows medical trainees to interact with it, enhancing its appeal as an educational resource. This study aims to investigate ChatGPT’s diagnostic accuracy and utility in medical education. Methods: 150 Medscape case challenges (September 2021 to January 2023) were inputted into ChatGPT. The primary outcome was the number (%) of cases for which the answer given was correct. Secondary outcomes included diagnostic accuracy, cognitive load, and quality of medical information. A qualitative content analysis was also conducted to assess its responses. Results: ChatGPT answered 49% (74/150) cases correctly. It had an overall accuracy of 74%, a precision of 48.67%, sensitivity of 48.67%, specificity of 82.89%, and an AUC of 0.66. Most answers were considered low cognitive load 51% (77/150) and most answers were complete and relevant 52% (78/150). Discussion: ChatGPT in its current form is not accurate as a diagnostic tool. ChatGPT does not necessarily give factual correctness, despite the vast amount of information it was trained on. Based on our qualitative analysis, ChatGPT struggles with the interpretation of laboratory values, imaging results, and may overlook key information relevant to the diagnosis. However, it still offers utility as an educational tool. ChatGPT was generally correct in ruling out a specific differential diagnosis and providing reasonable next diagnostic steps. Additionally, answers were easy to understand, showcasing a potential benefit in simplifying complex concepts for medical learners. Our results should guide future research into harnessing ChatGPT’s potential educational benefits, such as simplifying medical concepts and offering guidance on differential diagnoses and next steps.Keywords
This publication has 34 references indexed in Scilit:
- Evaluating the Feasibility of ChatGPT in Healthcare: An Analysis of Multiple Clinical and Research ScenariosJournal of Medical Systems, 2023
- Exploring ChatGPT for information of cardiopulmonary resuscitationResuscitation, 2023
- ChatGPT: five priorities for researchNature, 2023
- Natural language processing: state of the art, current trends and challengesMultimedia Tools and Applications, 2022
- Artificial Intelligence in Health CareJAMA, 2020
- Reimagining Medical Education in the Age of AIAMA Journal of Ethics, 2019
- A guide to deep learning in healthcareNature Medicine, 2019
- Clinical Decision Support in the Era of Artificial IntelligenceJAMA, 2018
- Artificial Intelligence in Medical Practice: The Question to the Answer?The American Journal of Medicine, 2018
- Extracting Information from Textual Documents in the Electronic Health Record: A Review of Recent ResearchYearbook of Medical Informatics, 2008