Chatbots' Performance in Premature Ejaculation Questions: A Comparative Analysis of Reliability, Readability, and Understandability
Authors:
- Serkan GonultasSGSerkan Gonultas
- Mücahit GelmişMGMücahit Gelmiş
- Mustafa Gökhan KöseMKMustafa Gökhan Köse
- Suhejb SulejmanSSSuhejb Sulejman
- Serhat YenturSYSerhat Yentur
- Burak ArslanBABurak Arslan
Preprint
- 20 March 2025
- preprint
- Published by Springer Nature in Research Square
Abstract
Objective This study aimed to evaluate the reliability, readability, and understandability of chatbot responses to frequently asked questions about premature ejaculation, and to assess the contributions, potential risks, and limitations of artificial intelligence. Methods Fifteen questions were selected using data from Google Trends and posed to the chatbots Copilot, Gemini, ChatGPT 4o, ChatGPT 4o Plus, and DeepSeek-R1. Reliability was evaluated using the Global Quality Scala (GQS) scale by two experts, readability was assessed with the Flesch Reading Ease (FRES), Flesch-Kincaid Grade Level (FKGL), Gunning Fog Index (GFI), and Simple Measure of Gobbledygook (SMOG) scales, and understandability was evaluated using the Patient Educational Materials Assessment Tool for Printable Materials (PEMAT-P) scale. Additionally, the consistency of source citations was examined. Results The GQS scores were as follows: Copilot: 3.96 ± 0.66, Gemini: 3.66 ± 0.78, ChatGPT 4o: 4.83 ± 0.23, ChatGPT 4o Plus: 4.83 ± 0.29, DeepSeek: 4.86 ± 0.22. The PEMAT-P scores were: Copilot: 0.70 ± 0.05, Gemini: 0.72 ± 0.04, ChatGPT 4o: 0.83 ± 0.03, ChatGPT 4o Plus: 0.77 ± 0.06, DeepSeek: 0.79 ± 0.06. While ChatGPT and DeepSeek scored higher for reliability and understandability, all chatbots performed at an acceptable level. However, readability scores were above the recommended level for the target audience. Instances of low reliability or unverified sources were noted, with no significant differences between the chatbots. Conclusion Chatbots provide highly reliable and informative responses regarding premature ejaculation; however, it is evident that there are significant limitations that require improvement, particularly concerning readability and the reliability of sources.Keywords
All Related Versions
This publication has 17 references indexed in Scilit:
- New technologies developed for treatment of premature ejaculationInternational Journal Of Impotence Research, 2024
- Assessment of Artificial Intelligence Chatbot Responses to Top Searched Queries About CancerJAMA Oncology, 2023
- Evaluating the performance of ChatGPT in answering questions related to urolithiasisInternational Urology and Nephrology, 2023
- Scientific discovery in the age of artificial intelligenceNature, 2023
- Artificial Intelligence and Machine Learning in Clinical Medicine, 2023New England Journal of Medicine, 2023
- Development of the Patient Education Materials Assessment Tool (PEMAT): A new measure of understandability and actionability for print and audiovisual patient informationPatient Education and Counseling, 2014
- The Premature Ejaculation Prevalence and Attitudes (PEPA) Survey: Prevalence, Comorbidities, and Professional Help-SeekingEuropean Urology, 2007
- Prevalence, Characteristics and Implications of Premature Ejaculation/Rapid EjaculationJournal of Urology, 2006
- Erectile Dysfunction and Premature Ejaculation are the Most Frequently Self-Reported Sexual Concerns: Profiles of 9,536 Men Calling A HelplineEuropean Urology, 2006
- Premature ejaculation: Is it a biogenic or a psychogenic disorder?Sexual and Marital Therapy, 1998