Assessing the Quality of AI Explanations of Atrial Fibrillation: Comparative Analysis of ChatGPT and Google Gemini (Preprint)
Preprint
- 10 March 2025
- preprint
- Published by JMIR Publications Inc. in JMIR Preprints
Abstract
BACKGROUND Atrial fibrillation (AF) is one of the most common arrhythmias and is associated with multiple risk factors, including stroke and heart failure. Therefore, it is important that accurate and comprehensive educational resources on AF are available. Artificial intelligence (AI) platforms such as Google Gemini and ChatGPT are emerging tools for medical education. To our knowledge, this is the first study to assess the quality of these platforms for AF education. OBJECTIVE This pilot study aimed to (1) assess the quality of ChatGPT and Google Gemini explanations of AF and its treatment, (2) compare responses from both platforms, and (3) analyze differences in interpretation between cardiologists and nonmedical professionals. METHODS The prompt “Explain atrial fibrillation and how to treat it to a patient” was entered into both ChatGPT and Google Gemini. The responses, along with a survey created in Google Forms, were sent out to potential participants between September and October 2024. The survey questions were derived from the Patient Education Materials Assessment Tool for Printable Materials and DISCERN criteria. Each survey question was multiple choice and rated on a 5-point Likert scale, where 1 indicated strong disagreement and 5 indicated strong agreement. Averages of each answer from the survey were calculated and compared using the Wilcoxon signed-rank test. RESULTS The survey was completed by 28 participants, including cardiologists and cardiology fellows and 17 nonmedical professionals (college graduates not employed in the medical field). No significant overall quality differences were observed between ChatGPT and Google Gemini. When comparing participant groups, cardiologists rated AI-generated responses higher than nonmedical professionals for perceived bias (mean 3.82 vs mean 4.33; P=.04) and for explaining consequences of no treatment (mean 2.85 vs mean 1.86; P=.005). In contrast, nonmedical professionals rated structural and organizational features more favorably than cardiologists, including visual cues (mean 3.97 vs mean 2.48; P<.001), informative headers (mean 4.77 vs mean 3.95; P=.001), concise sections (mean 4.71 vs mean 4.29; P=.04), identifiable patient actions (mean 4.21 vs mean 3.19; P=.04), and references to additional resources (mean 2.65 vs mean 1.71; P=.006). CONCLUSIONS The comparable quality of ChatGPT and Google Gemini suggests that both are viable for AF education. Differences in ratings of organizational features versus perceptions of bias and consequences of no treatment highlight a disconnect between cardiologists and nonmedical professionals. To optimize clinical application, cardiologists should evaluate both comprehensiveness and clarity of AI-generated responses. This serves as a pilot study, and future research should focus on longitudinal studies of AI-generated responses and their effects on clinical outcomes.Keywords
All Related Versions
- Published version: JMIR Formative Research, 10, e73715.
This publication has 20 references indexed in Scilit:
- Artificial-intelligence-based risk prediction and mechanism discovery for atrial fibrillation using heart beat-to-beat intervalsMed, 2024
- 2023 ACC/AHA/ACCP/HRS Guideline for the Diagnosis and Management of Atrial Fibrillation: A Report of the American College of Cardiology/American Heart Association Joint Committee on Clinical Practice GuidelinesCirculation, 2024
- Accuracy and comprehensibility of chat-based artificial intelligence for patient information on atrial fibrillation and cardiac implantable electronic devicesEP Europace, 2023
- ChatGPT's Response to the Diabetes Knowledge Questionnaire: Implications for Diabetes EducationDiabetes Technology & Therapeutics, 2023
- Artificial intelligence-guided screening for atrial fibrillation using electrocardiogram during sinus rhythm: a prospective non-randomised interventional trialThe Lancet, 2022
- Artificial intelligence for a personalized diagnosis and treatment of atrial fibrillationAmerican Journal of Physiology-Heart and Circulatory Physiology, 2021
- Comparison of an Artificial Intelligence–Enabled Patient Decision Aid vs Educational Material on Decision Quality, Shared Decision-Making, Patient Experience, and Functional Outcomes in Adults With Knee OsteoarthritisJAMA Network Open, 2021
- The effectiveness of psychological interventions on self-care, psychological and health outcomes in patients with chronic heart failure—A systematic review and meta-analysisInternational Journal of Nursing Studies, 2018
- Patient Education Materials Assessment Tool2014
- DISCERN: an instrument for judging the quality of written consumer health information on treatment choices.Journal of Epidemiology and Community Health, 1999