ChatGPT as an information tool in rhinology. Can we trust each other today?
- 4 March 2024
- journal article
- research article
- Published by Springer Nature in European Archives of Oto-Rhino-Laryngology
- Vol. 281 (6) , 3253-3259
Abstract
Purpose ChatGPT (Chat-Generative Pre-trained Transformer) has proven to be a powerful information tool on various topics, including healthcare. This system is based on information obtained on the Internet, but this information is not always reliable. Currently, few studies analyze the validity of these responses in rhinology. Our work aims to assess the quality and reliability of the information provided by AI regarding the main rhinological pathologies. Methods We asked to the default ChatGPT version (GPT-3.5) 65 questions about the most prevalent pathologies in rhinology. The focus was learning about the causes, risk factors, treatments, prognosis, and outcomes. We use the Discern questionnaire and a hexagonal radar schema to evaluate the quality of the information. We use Fleiss's kappa statistical analysis to determine the consistency of agreement between different observers. Results The overall evaluation of the Discern questionnaire resulted in a score of 4.05 (± 0.6). The results in the Reliability section are worse, with an average score of 3.18. (± 1.77). This score is affected by the responses to questions about the source of the information provided. The average score for the Quality section was 3.59 (± 1.18). Fleiss's Kappa shows substantial agreement, with a K of 0.69 (p < 0.001). Conclusion The ChatGPT answers are accurate and reliable. It generates a simple and understandable description of the pathology for the patient's benefit. Our team considers that ChatGPT could be a useful tool to provide information under prior supervision by a health professional.Keywords
This publication has 16 references indexed in Scilit:
- Validity and reliability of an instrument evaluating the performance of intelligent chatbot: the Artificial Intelligence Performance Instrument (AIPI)European Archives of Oto-Rhino-Laryngology, 2023
- Validity of the large language model ChatGPT (GPT4) as a patient information source in otolaryngology by a variety of doctors in a tertiary otorhinolaryngology departmentActa Oto-Laryngologica, 2023
- Assessing the accuracy of ChatGPT references in head and neck and ENT disciplinesEuropean Archives of Oto-Rhino-Laryngology, 2023
- ChatGPT and most frequent urological diseases: analysing the quality of information and potential risks for patientsWorld Journal of Urology, 2023
- BPPV Information on Google Versus AI (ChatGPT)Otolaryngology–Head and Neck Surgery, 2023
- ChatGPT and large language model (LLM) chatbots: The current state of acceptability and a proposal for guidelines on utilization in academic medicineJournal of Pediatric Urology, 2023
- Role of Chat GPT in Public HealthAnnals of Biomedical Engineering, 2023
- Position Paper on Nasal Obstruction: Evaluation and TreatmentJournal of Investigational Allergology and Clinical Immunology, 2018
- Endonasal Endoscopic Surgery in the Management of Sinonasal and Anterior Skull Base MalignanciesHead and Neck Pathology, 2016
- DISCERN: an instrument for judging the quality of written consumer health information on treatment choices.Journal of Epidemiology and Community Health, 1999