Artificial intelligence in obstetrics and gynecology: Evaluating ChatGPT and Google Gemini in answering patient questions
Open Access
- 28 October 2025
- journal article
- Published by Wiley in International Journal of Gynecology & Obstetrics
- Vol. 173 (1) , 328-335
Abstract
Introduction: To evaluate the accuracy and completeness of responses across common obstetrical and gynecologic topics generated by the large language models (LLMs) ChatGPT and Google Gemini, which have become increasingly popular for patients seeking medical information before physician consultations. Methods: Ten topics were identified, five obstetrical (prenatal labs, extended carrier screen, treatments for nausea and vomiting in pregnancy, gestational diabetes, and trial of labor after cesarean section) and five gynecologic (polycystic ovary syndrome, pelvic inflammatory disease, cervical smears, mammograms, and birth control). For each condition, ChatGPT generated five of the most frequently asked patient questions, which were then presented separately to ChatGPT and Google Gemini. Board‐certified Obstetrics and Gynecology physicians evaluated the responses using Likert scales for accuracy (1–6) and completeness (1–3). Results: Acceptable response criteria were defined as an accuracy score of 5 or greater (“nearly all correct”) and a completeness score of 2 or greater (“adequately complete”). Most responses from both models met these thresholds. Wilcoxon signed‐rank tests demonstrated statistically significant differences in accuracy and completeness between models ( P < 0.05). Inter‐rater agreement was measured using intraclass correlation coefficients. For obstetrical topics, ChatGPT scored −0.047 (completeness) and 0.112 (accuracy), whereas Google Gemini scored 0.367 and 0.205, respectively. For gynecologic topics, ChatGPT scored 0.328 and 0.20, compared with Google Gemini at 0.151 and −0.08. Conclusion: Both LLMs provided largely accurate and complete responses to patient questions. ChatGPT demonstrated stronger outcomes overall, suggesting potential utility in patient education; however, patients should confirm online information with physicians given the limitations of LLMs.Keywords
This publication has 24 references indexed in Scilit:
- Accuracy of machine learning and traditional statistical models in the prediction of postpartum haemorrhage: a systematic reviewBMJ Open, 2025
- Enhancing perinatal health patient information through ChatGPT – An accuracy studyPEC Innovation, 2025
- Exploring the Efficacy of Artificial Intelligence: A Comprehensive Analysis of CHAT‐GPT's Accuracy and Completeness in Addressing Urinary Incontinence QueriesNeurourology and Urodynamics, 2024
- The doc versus the bot: A pilot study to assess the quality and accuracy of physician and chatbot responses to clinical questions in gynecologic oncologyGynecologic Oncology Reports, 2024
- Exploring the capabilities of ChatGPT in women’s health: obstetrics and gynaecologynpj Women's Health, 2024
- Evaluation of the reliability and readability of ChatGPT-4 responses regarding hypothyroidism during pregnancyScientific Reports, 2024
- Evaluation of Online Artificial Intelligence-Generated Information on Common Hand ProceduresThe Journal of Hand Surgery, 2023
- Transcending Language Barriers in Obstetrics and GynecologyObstetrics & Gynecology, 2023
- Abstracts written by ChatGPT fool scientistsNature, 2023
- Widespread adoption of AI risk models for maternal hemorrhage requires mitigation of biasAmerican Journal of Obstetrics and Gynecology, 2023