Chatbots’ performance in premature ejaculation questions: a comparative analysis of reliability, readability, and understandability
- 24 September 2025
- journal article
- research article
- Published by Springer Nature in International Journal Of Impotence Research
- Vol. 37 (10) , 843-849
Abstract
This study aimed to evaluate the reliability, readability, and understandability of chatbot responses to frequently asked questions about premature ejaculation, and to assess the contributions, potential risks, and limitations of artificial intelligence. Fifteen questions were selected using data from Google Trends and posed to the chatbots Copilot, Gemini, ChatGPT4o, ChatGPT4oPlus, and DeepSeek-R1. Reliability was evaluated using the Global Quality Scale(GQS) by two experts, readability was assessed with the Flesch Kincaid Reading Ease(FKRE), Flesch Kincaid Grade Level(FKGL), Gunning Fog Index(GFI), and Simple Measure of Gobbledygook(SMOG), and understandability was evaluated using the Patient Educational Materials Assessment Tool for Printable Materials(PEMAT-P). Additionally, the consistency of source citations was examined. The GQS were as follows: Copilot: 3.96 ± 0.66, Gemini: 3.66 ± 0.78, ChatGPT4o: 4.83 ± 0.23, ChatGPT4oPlus: 4.83 ± 0.29, DeepSeek-R1:4.86 ± 0.22 (p < 0.001). The PEMAT-P were as follows: Copilot: 0.70 ± 0.05, Gemini: 0.72 ± 0.04, ChatGPT4o: 0.83 ± 0.03, ChatGPT4oPlus: 0.77 ± 0.06, DeepSeek-R1:0.79 ± 0.06 (p < 0.001). While ChatGPT4oPlus and DeepSeek-R1 scored higher for reliability and understandability, all chatbots performed at an acceptable level (≥70%). However, readability scores were above the recommended level for the target audience. Instances of low reliability or unverified sources were noted, with no significant differences between the chatbots. Chatbots provide highly reliable and informative responses regarding premature ejaculation; however, it is evident that there are significant limitations that require improvement, particularly concerning readability and the reliability of sources.Keywords
All Related Versions
This publication has 19 references indexed in Scilit:
- Artificial Intelligence in Andrology: A New Frontier in Male Infertility Diagnosis and TreatmentCurrent Urology Reports, 2025
- Frequently asked questions on erectile dysfunction: evaluating artificial intelligence answers with expert mentorshipInternational Journal Of Impotence Research, 2024
- New technologies developed for treatment of premature ejaculationInternational Journal Of Impotence Research, 2024
- Scientific discovery in the age of artificial intelligenceNature, 2023
- Artificial Intelligence and Machine Learning in Clinical Medicine, 2023New England Journal of Medicine, 2023
- Applications of artificial intelligence in the diagnosis and prediction of erectile dysfunction: a narrative reviewInternational Journal Of Impotence Research, 2022
- Development of the Patient Education Materials Assessment Tool (PEMAT): A new measure of understandability and actionability for print and audiovisual patient informationPatient Education and Counseling, 2014
- The Premature Ejaculation Prevalence and Attitudes (PEPA) Survey: Prevalence, Comorbidities, and Professional Help-SeekingEuropean Urology, 2007
- Prevalence, Characteristics and Implications of Premature Ejaculation/Rapid EjaculationJournal of Urology, 2006
- Erectile Dysfunction and Premature Ejaculation are the Most Frequently Self-Reported Sexual Concerns: Profiles of 9,536 Men Calling A HelplineEuropean Urology, 2006