ChatGPT’s adherence to otolaryngology clinical practice guidelines
Authors:
- 22 April 2024
- journal article
- research article
- Published by Springer Nature in European Archives of Oto-Rhino-Laryngology
- Vol. 281 (7) , 3829-3834
Abstract
Objectives Large language models, including ChatGPT, has the potential to transform the way we approach medical knowledge, yet accuracy in clinical topics is critical. Here we assessed ChatGPT’s performance in adhering to the American Academy of Otolaryngology-Head and Neck Surgery guidelines. Methods We presented ChatGPT with 24 clinical otolaryngology questions based on the guidelines of the American Academy of Otolaryngology. This was done three times (N = 72) to test the model’s consistency. Two otolaryngologists evaluated the responses for accuracy and relevance to the guidelines. Cohen’s Kappa was used to measure evaluator agreement, and Cronbach’s alpha assessed the consistency of ChatGPT’s responses. Results The study revealed mixed results; 59.7% (43/72) of ChatGPT’s responses were highly accurate, while only 2.8% (2/72) directly contradicted the guidelines. The model showed 100% accuracy in Head and Neck, but lower accuracy in Rhinology and Otology/Neurotology (66%), Laryngology (50%), and Pediatrics (8%). The model’s responses were consistent in 17/24 (70.8%), with a Cronbach’s alpha value of 0.87, indicating a reasonable consistency across tests. Conclusions Using a guideline-based set of structured questions, ChatGPT demonstrates consistency but variable accuracy in otolaryngology. Its lower performance in some areas, especially Pediatrics, suggests that further rigorous evaluation is needed before considering real-world clinical use.Keywords
This publication has 27 references indexed in Scilit:
- Exploring the role of ChatGPT in clinical decision-making in otorhinolaryngology: a ChatGPT designed studyEuropean Archives of Oto-Rhino-Laryngology, 2024
- ChatGPT‐4 performance in rhinology: A clinical case seriesInternational Forum of Allergy & Rhinology, 2024
- Evaluating the Current Ability of ChatGPT to Assist in Professional Otolaryngology EducationOTO Open, 2023
- Accuracy of ChatGPT‐Generated Information on Head and Neck and Oromaxillofacial Surgery: A Multicenter Collaborative AnalysisOtolaryngology–Head and Neck Surgery, 2023
- Sailing the Seven Seas: A Multinational Comparison of ChatGPT’s Performance on Medical Licensing ExaminationsAnnals of Biomedical Engineering, 2023
- Diagnostic and Management Applications of ChatGPT in Structured Otolaryngology Clinical ScenariosOTO Open, 2023
- ChatGPT’s quiz skills in different otolaryngology subspecialties: an analysis of 2576 single-choice and multiple-choice board certification preparation questionsEuropean Archives of Oto-Rhino-Laryngology, 2023
- Open artificial intelligence platforms in nursing education: Tools for academic progress or abuse?Nurse Education in Practice, 2022
- Daily briefing: Will ChatGPT kill the essay assignment?Nature, 2022
- Are ChatGPT and AlphaCode going to replace programmers?Nature, 2022