ChatGPT‐4 Performs Clinical Information Retrieval Tasks Using Consistently More Trustworthy Resources Than Does Google Search for Queries Concerning the Latarjet Procedure
Authors:
- 25 June 2024
- journal article
- research article
- Published by Wiley in Arthroscopy
- Vol. 41 (3) , 588-597
Abstract
Purpose: To assess the ability of ChatGPT‐4, an automated Chatbot powered by artificial intelligence, to answer common patient questions concerning the Latarjet procedure for patients with anterior shoulder instability and compare this performance with Google Search Engine. Methods: Using previously validated methods, a Google search was first performed using the query “Latarjet.” Subsequently, the top 10 frequently asked questions (FAQs) and associated sources were extracted. ChatGPT‐4 was then prompted to provide the top 10 FAQs and answers concerning the procedure. This process was repeated to identify additional FAQs requiring discrete‐numeric answers to allow for a comparison between ChatGPT‐4 and Google. Discrete, numeric answers were subsequently assessed for accuracy on the basis of the clinical judgment of 2 fellowship‐trained sports medicine surgeons who were blinded to search platform. Results: Mean (± standard deviation) accuracy to numeric‐based answers was 2.9 ± 0.9 for ChatGPT‐4 versus 2.5 ± 1.4 for Google ( P = .65). ChatGPT‐4 derived information for answers only from academic sources, which was significantly different from Google Search Engine ( P = .003), which used only 30% academic sources and websites from individual surgeons (50%) and larger medical practices (20%). For general FAQs, 40% of FAQs were found to be identical when comparing ChatGPT‐4 and Google Search Engine. In terms of sources used to answer these questions, ChatGPT‐4 again used 100% academic resources, whereas Google Search Engine used 60% academic resources, 20% surgeon personal websites, and 20% medical practices ( P = .087). Conclusions: ChatGPT‐4 demonstrated the ability to provide accurate and reliable information about the Latarjet procedure in response to patient queries, using multiple academic sources in all cases. This was in contrast to Google Search Engine, which more frequently used single‐surgeon and large medical practice websites. Despite differences in the resources accessed to perform information retrieval tasks, the clinical relevance and accuracy of information provided did not significantly differ between ChatGPT‐4 and Google Search Engine. Clinical Relevance: Commercially available large language models (LLMs), such as ChatGPT‐4, can perform diverse information retrieval tasks on‐demand. An important medical information retrieval application for LLMs consists of the ability to provide comprehensive, relevant, and accurate information for various use cases such as investigation about a recently diagnosed medical condition or procedure. Understanding the performance and abilities of LLMs for use cases has important implications for deployment within health care settings.Keywords
Funding Information
- American Association of Orthopaedic Surgeons
- Smith and Nephew
- Stryker
- Zimmer
This publication has 18 references indexed in Scilit:
- Readability of online patient education materials for shoulder instability surgery in English and SpanishJournal of Shoulder and Elbow Surgery, 2024
- Accuracy and Reliability of Chatbot Responses to Physician QuestionsJAMA Network Open, 2023
- Benchmarking large language models’ performances for myopia care: a comparative analysis of ChatGPT-3.5, ChatGPT-4.0, and Google BardEBioMedicine, 2023
- Exploring the potential of ChatGPT as a supplementary tool for providing orthopaedic informationKnee Surgery, Sports Traumatology, Arthroscopy, 2023
- What’s all the chatter about?The Bone & Joint Journal, 2023
- Using a Google Web Search Analysis to Assess the Utility of ChatGPT in Total Joint ArthroplastyThe Journal of Arthroplasty, 2023
- A quality assessment of YouTube content on shoulder instabilityThe Physician and Sportsmedicine, 2021
- Quality of Online Video Resources Concerning Patient Education for the Meniscus: A YouTube‐Based Quality‐Control StudyArthroscopy, 2020
- International assessment on quality and content of internet information on osteoarthritisOsteoarthritis and Cartilage, 2018
- Interventions to prevent and reduce physician burnout: a systematic review and meta-analysisThe Lancet, 2016