Can ChatGPT-5 educate the public about vasectomy?: a Google Trends–based expert panel assessment
Authors:
Open Access
- 18 March 2026
- journal article
- research article
- Published by Frontiers Media SA in Frontiers in Digital Health
- Vol. 8, 1726517
Abstract
Background: ChatGPT-5, the latest multimodal large language model (LLM), has gained remarkable public attention for its ability to provide real-time and context-aware health information. However, its effectiveness in addressing sensitive urological topics such as vasectomy has not been systematically evaluated. Objective: This study aimed to evaluate the accuracy, completeness and public suitability of ChatGPT-5's responses to frequently asked questions about vasectomy, derived from Google Trends data reflecting real-world public interest. Methods: A total of eight experts—four urologists, two public health specialists, one obstetrician-gynecologist and one fertility nurse—independently assessed ChatGPT-5's responses to ten high-frequency vasectomy-related questions. Each response was rated using six 5-point Likert-scale criteria: medical accuracy, completeness, clarity, tone, public usefulness and recommendability. Descriptive statistics, Kruskal–Wallis tests and two-way random-effects intraclass correlation coefficients (ICC, 95% CI) were applied for statistical analysis. Results: The mean ratings across evaluation domains ranged from 3.75 to 4.04. Clarity of language and tone appropriateness received the highest scores, whereas medical accuracy and comprehensiveness demonstrated greater dispersion. No statistically significant differences were observed among expert subgroups ( p > 0.05). Inter-rater reliability was very low (ICC = −0.01), indicating substantial variability across expert evaluations. Conclusions: In this exploratory assessment, ChatGPT-5 responses to vasectomy-related public questions were frequently perceived as clear and appropriately framed for informational use. However, variability across expert ratings and the absence of layperson validation underscore the need for cautious interpretation. Large language model outputs may serve as supportive educational resources when accompanied by expert oversight and audience-specific adaptation.Keywords
This publication has 27 references indexed in Scilit:
- Artificial Intelligence-Based Large Language Models Can Facilitate Patient EducationJournal of the Pediatric Orthopaedic Society of North America, 2025
- Evaluating ChatGPT as a patient resource for frequently asked questions about lung cancer surgery—a pilot studyThe Journal of Thoracic and Cardiovascular Surgery, 2024
- Physician views of artificial intelligence in otolaryngology and rhinology: A mixed methods studyLaryngoscope Investigative Otolaryngology, 2023
- Caution! AI Bot Has Entered the Patient Chat: ChatGPT Has Limitations in Providing Accurate Urologic Healthcare AdviceUrology, 2023
- Evaluating the Feasibility of ChatGPT in Healthcare: An Analysis of Multiple Clinical and Research ScenariosJournal of Medical Systems, 2023
- Life Cycle Assessment of Reusable and Disposable Cystoscopes: A Path to Greener Urological ProceduresEuropean Urology Focus, 2022
- Report of the National Cancer Institute and the Eunice Kennedy Shriver National Institute of Child Health and Human Development–sponsored workshop: gynecology and women’s health–benign conditions and cancerAmerican Journal of Obstetrics and Gynecology, 2020
- A Guideline of Selecting and Reporting Intraclass Correlation Coefficients for Reliability ResearchJournal of Chiropractic Medicine, 2016
- Canadian Contraception Consensus (Part 1 of 4)Journal of Obstetrics and Gynaecology Canada, 2015
- Racial Differences in Vasectomy Utilization in the United States: Data From the National Survey of Family GrowthUrology, 2009