Exploring the Efficacy of Artificial Intelligence: A Comprehensive Analysis of CHAT‐GPT's Accuracy and Completeness in Addressing Urinary Incontinence Queries
- 10 October 2024
- journal article
- research article
- Published by Wiley in Neurourology and Urodynamics
- Vol. 44 (1) , 153-164
Abstract
Background: Artificial intelligence models are increasingly gaining popularity among patients and healthcare professionals. While it is impossible to restrict patient's access to different sources of information on the Internet, healthcare professional needs to be aware of the content‐quality available across different platforms.Objective: To investigate the accuracy and completeness of Chat Generative Pretrained Transformer (ChatGPT) in addressing frequently asked questions related to the management and treatment of female urinary incontinence (UI), compared to recommendations from guidelines.Methods: This is a cross‐sectional study. Two researchers developed 14 frequently asked questions related to UI. Then, they were inserted into the ChatGPT platform on September 16, 2023. The accuracy (scores from 1 to 5) and completeness (score from 1 to 3) of ChatGPT's answers were assessed individually by two experienced researchers in the Women's Health field, following the recommendations proposed by the guidelines for UI.Results: Most of the answers were classified as “more correct than incorrect” (n = 6), followed by “incorrect information than correct” (n = 3), “approximately equal correct and incorrect” (n = 2), “near all correct” (n = 2, and “correct” (n = 1). Regarding the appropriateness, most of the answers were classified as adequate, as they provided the minimum information expected to be classified as correct.Conclusion: These results showed an inconsistency when evaluating the accuracy of answers generated by ChatGPT compared by scientific guidelines. Almost all the answers did not bring the complete content expected or reported in previous guidelines, which highlights to healthcare professionals and scientific community a concern about using artificial intelligence in patient counseling.Keywords
This publication has 38 references indexed in Scilit:
- Performance of an Artificial Intelligence Chatbot in Ophthalmic Knowledge AssessmentJAMA Ophthalmology, 2023
- Assessing the Accuracy of Responses by the Language Model ChatGPT to Questions Regarding Bariatric SurgeryObesity Surgery, 2023
- Appropriateness of Cardiovascular Disease Prevention Recommendations Obtained From a Popular Online Chat-Based Artificial Intelligence ModelJAMA, 2023
- Urinary incontinence and quality of life: a systematic review and meta-analysisAging Clinical and Experimental Research, 2020
- Electrical stimulation with non-implanted devices for stress urinary incontinence in womenCochrane Database of Systematic Reviews, 2017
- An International Urogynecological Association (IUGA)/International Continence Society (ICS) joint report on the terminology for the conservative and nonpharmacological management of female pelvic floor dysfunctionNeurourology and Urodynamics, 2016
- More Than One-Third of US Individuals Use the Internet to Self-diagnoseJAMA, 2013
- Feedback or biofeedback to augment pelvic floor muscle training for urinary incontinence in womenCochrane Database of Systematic Reviews, 2011
- An International Urogynecological Association (IUGA)/International Continence Society (ICS) joint report on the terminology for female pelvic floor dysfunctionInternational Urogynecology Journal, 2009
- Needle emg registration of striated urethral wall and pelvic floor muscle activity patterns during cough, valsalva, abdominal, hip adductor, and gluteal muscle contractions in nulliparous healthy femalesNeurourology and Urodynamics, 1994