Readability, reliability and quality of responses generated by ChatGPT, gemini, and perplexity for the most frequently asked questions about pain
Top Cited Papers
Open Access
- 14 March 2025
- journal article
- research article
- Published by Wolters Kluwer Health in Medicine
- Vol. 104 (11) , e41780
Abstract
It is clear that artificial intelligence-based chatbots will be popular applications in the field of healthcare in the near future. It is known that more than 30% of the world’s population suffers from chronic pain and individuals try to access the health information they need through online platforms before applying to the hospital. This study aimed to examine the readability, reliability and quality of the responses given by 3 different artificial intelligence chatbots (ChatGPT, Gemini and Perplexity) to frequently asked questions about pain. In this study, the 25 most frequently used keywords related to pain were determined using Google Trend and asked to every 3 artificial intelligence chatbots. The readability of the response texts was determined by Flesch Reading Ease Score (FRES), Simple Measure of Gobbledygook, Gunning Fog and Flesch-Kincaid Grade Level readability scoring. Reliability assessment was determined by the Journal of American Medical Association (JAMA), DISCERN scales. Global Quality Score and Ensuring Quality Information for Patients (EQIP) score were used in quality assessment. As a result of Google Trend search, the first 3 keywords were determined as “back pain,” “stomach pain,” and “chest pain.” The readability of the answers given by all 3 artificial intelligence applications was determined to be higher than the recommended 6th grade readability level ( P < .001). In the readability evaluation, the order from easy to difficult was determined as Google Gemini, ChatGPT and Perplexity. Higher GQS scores ( P = .008) were detected in Gemini compared to other chatbots. Perplexity had higher JAMA, DISCERN and EQIP scores compared to other chatbots, respectively ( P < .001, P < .001, P < .05). It has been determined that the answers given by ChatGPT, Gemini, and Perplexity to pain-related questions are difficult to read and their reliability and quality are low. It can be stated that these artificial intelligence chatbots cannot replace a comprehensive medical consultation. In artificial intelligence applications, it may be recommended to facilitate the readability of text content, create texts containing reliable references, and control them by a supervisory expert team.Keywords
This publication has 40 references indexed in Scilit:
- Responses of Five Different Artificial Intelligence Chatbots to the Top Searched Queries About Erectile Dysfunction: A Comparative AnalysisJournal of Medical Systems, 2024
- Chatbot Performance in Defining and Differentiating Palliative Care, Supportive Care, Hospice CareJournal of Pain and Symptom Management, 2024
- Assessment of Artificial Intelligence Chatbot Responses to Top Searched Queries About CancerJAMA Oncology, 2023
- How Well Do Artificial Intelligence Chatbots Respond to the Top Search Queries About Urological Malignancies?European Urology, 2023
- Appropriateness and Readability of ChatGPT-4-Generated Responses for Surgical Treatment of Retinal DiseasesOphthalmology Retina, 2023
- A deeper dive into ChatGPT: history, use and future perspectives for orthopaedic researchKnee Surgery, Sports Traumatology, Arthroscopy, 2023
- Readability assessment of online peripheral artery disease education materialsJournal of Vascular Surgery, 2022
- Evaluation of Online Written Medication Educational Resources for People Living With Heart FailureCJC Open, 2022
- Artificial intelligence in medicineMetabolism, 2017
- Infodemiology: the epidemiology of (mis)informationThe American Journal of Medicine, 2002