Evaluation of the accuracy of ChatGPT in answering asthma-related questions
Authors:
- Bruno Pellozo Cerqueira1BCBruno Pellozo Cerqueira1
- Vinicius Cappellette da Silva Leite1VLVinicius Cappellette da Silva Leite1
- Carla Gonzaga França1CFCarla Gonzaga França1
- Fernando Sergio Leitão Filho2FFFernando Sergio Leitão Filho2
- Sonia Maria Faresin2SFSonia Maria Faresin2
- Ricardo Gassmann Figueiredo3RFRicardo Gassmann Figueiredo3
- Andrea Antunes Cetlin4ACAndrea Antunes Cetlin4
- Lilian Serrasqueiro Ballini Caetano2LCLilian Serrasqueiro Ballini Caetano2
- José Baddini-Martinez2JBJosé Baddini-Martinez2
Open Access
- 16 June 2025
- journal article
- research article
- Published by Sociedade Brasileira de Pneumologia e Tisiologia in Jornal Brasileiro de Pneumologia
- Vol. 51 (3) , e20240388
Abstract
Objective: To evaluate the quality of ChatGPT answers to asthma-related questions, as assessed from the perspectives of asthma specialists and laypersons. Methods: Seven asthma-related questions were asked to ChatGPT (version 4) between May 3, 2024 and May 4, 2024. The questions were standardized with no memory of previous conversations to avoid bias. Six pulmonologists with extensive expertise in asthma acted as judges, independently assessing the quality and reproducibility of the answers from the perspectives of asthma specialists and laypersons. A Likert scale ranging from 1 to 4 was used, and the content validity coefficient was calculated to assess the level of agreement among the judges. Results: The evaluations showed variability in the quality of the answers provided by ChatGPT. From the perspective of asthma specialists, the scores ranged from 2 to 3, with greater divergence in questions 2, 3, and 5. From the perspective of laypersons, the content validity coefficient exceeded 0.80 for four of the seven questions, with most answers being correct despite a lack of significant depth. Conclusions: Although ChatGPT performed well in providing answers to laypersons, the answers that it provided to specialists were less accurate and superficial. Although AI has the potential to provide useful information to the public, it should not replace medical guidance. Critical analysis of AI-generated information remains essential for health care professionals and laypersons alike, especially for complex conditions such as asthma.Keywords
This publication has 14 references indexed in Scilit:
- Introducing ChatGPT for Python Developers2024
- Quality of Chatbot Information Related to Benign Prostatic HyperplasiaThe Prostate, 2024
- Evaluating the scientific reliability of ChatGPT as a source of information on asthmaJournal of Allergy and Clinical Immunology: Global, 2024
- Comparing Physician and Artificial Intelligence Chatbot Responses to Patient Questions Posted to a Public Social Media ForumJAMA Internal Medicine, 2023
- User Intentions to Use ChatGPT for Self-Diagnosis and Health-Related Purposes: Cross-sectional Survey StudyJMIR Human Factors, 2023
- Artificial Intelligence: Definition and Background2023
- “International ERS/ATS guidelines on definition, evaluation and treatment of severe asthma.” Kian Fan Chung, Sally E. Wenzel, Jan L. Brozek,et al. Eur Respir J2014; 43: 343–373.European Respiratory Journal, 2022
- Heterogeneity within and between physician-diagnosed asthma and/or COPD: NOVELTY cohortEuropean Respiratory Journal, 2021
- History of artificial intelligence in medicineGastrointestinal Endoscopy, 2020
- AsthmaThe Lancet, 2018