Advancement of Generative Pre-trained Transformer Chatbots in Answering Clinical Questions in the Practical Rhinoplasty Guideline
- 25 September 2024
- journal article
- research article
- Published by Springer Nature in Aesthetic Plastic Surgery
- Vol. 49 (7) , 1874-1880
Abstract
Background The Generative Pre-trained Transformer (GPT) series, which includes ChatGPT, is an artificial large language model that provides human-like text dialogue. This study aimed to evaluate the performance of artificial intelligence chatbots in answering clinical questions based on practical rhinoplasty guidelines. Methods Clinical questions (CQs) developed from the guidelines were used as question sources. For each question, we asked GPT-4 and GPT-3.5 (ChatGPT), developed by OpenAI, to provide answers for the CQs, Policy Level, Aggregate Evidence Quality, Level of Confidence in Evidence, and References. We compared the performance of the two types of artificial intelligence (AI) chatbots. Results A total of 10 questions were included in the final analysis, and the AI chatbots correctly answered 90.0% of these. GPT-4 demonstrated a lower accuracy rate than GPT-3.5 in answering CQs, although without statistically significant difference (86.0% vs. 94.0%; p = 0.05), whereas GPT-4 showed significantly higher accuracy for the level of confidence in Evidence than GPT-3.5 (52.0% vs. 28.0%; p < 0.01). No statistical differences were observed in Policy Level, Aggregate Evidence Quality, and Reference Match. In addition, GPT-4 rated significantly higher in presenting existing references than GPT-3.5 (36.9% vs. 24.1%; p = 0.01). Conclusions The overall performance of GPT-4 was similar to that of GPT-3.5. However, GPT-4 provided existing references at a higher rate than GPT-3.5. GPT-4 has the potential to provide a more accurate reference in professional fields, including rhinoplasty. Level of Evidence V This journal requires that authors assign a level of evidence to each article. For a full description of these Evidence-Based Medicine ratings, please refer to the Table of Contents or the online Instructions to Authors www.springer.com/00266.Keywords
This publication has 38 references indexed in Scilit:
- Deep Learning for Classification of Pediatric Otitis MediaThe Laryngoscope, 2020
- Applied Deep Learning in Plastic Surgery: Classifying Rhinoplasty With a Mobile AppJournal of Craniofacial Surgery, 2020
- Making the Subjective Objective: Machine Learning and RhinoplastyAesthetic Surgery Journal, 2019
- A guide to deep learning in healthcareNature Medicine, 2019
- Clinical Practice Guideline: Improving Nasal Form and Function after RhinoplastyOtolaryngology–Head and Neck Surgery, 2017
- The First Smartphone Application for Microsurgery MonitoringPlastic and Reconstructive Surgery, 2014
- The human external nose and its evolutionary role in the prevention of obstructive sleep apneaOtolaryngology–Head and Neck Surgery, 2010
- Dynamics in RhinoplastyPlastic and Reconstructive Surgery, 2000
- Psychosocial consequences of nasal aesthetic and functional surgery: a controlled prospective study in an ENT setting.1998
- Interactive Computer Graphics: A New Technology to Improve Judgment in RhinoplastyClinics in Plastic Surgery, 1987