ChatGPT Provides Satisfactory but Occasionally Inaccurate Answers to Common Patient Hip Arthroscopy Questions
Authors:
Open Access
- 22 June 2024
- journal article
- Published by Wiley in Arthroscopy
- Vol. 41 (5) , 1337-1347
Abstract
Purpose: To assess the ability of ChatGPT to answer common patient questions regarding hip arthroscopy, and to analyze the accuracy and appropriateness of its responses. Methods: Ten questions were selected from well‐known patient education websites, and ChatGPT (version 3.5) responses to these questions were graded by 2 fellowship‐trained hip preservation surgeons. Responses were analyzed, compared with the current literature, and graded from A to D (A being the highest, and D being the lowest) in a grading scale on the basis of the accuracy and completeness of the response. If the grading differed between the 2 surgeons, a consensus was reached. Inter‐rater agreement was calculated. The readability of responses was also assessed using the Flesch‐Kincaid Reading Ease Score (FRES) and Flesch‐Kincaid Grade Level (FKGL). Results: Responses received the following consensus grades: A (50%, n = 5), B (30%, n = 3), C (10%, n = 1), D (10%, n = 1). Inter‐rater agreement on the basis of initial individual grading was 30%. The mean FRES was 28.2 (± 9.2 standard deviation), corresponding to a college graduate level, ranging from 11.7 to 42.5. The mean FKGL was 14.4 (±1.8 standard deviation), ranging from 12.1 to 18, indicating a college student reading level. Conclusions: ChatGPT can answer common patient questions regarding hip arthroscopy with satisfactory accuracy graded by 2 high‐volume hip arthroscopists; however, incorrect information was identified in more than one instance. Caution must be observed when using ChatGPT for patient education related to hip arthroscopy. Clinical Relevance: Given the increasing number of hip arthroscopies being performed annually, ChatGPT has the potential to aid physicians in educating their patients about this procedure and addressing any questions they may have.Keywords
This publication has 46 references indexed in Scilit:
- Two minutes of orthopaedics with ChatGPT: it is just the beginning; it’s going to be hot, hot, hot!International Orthopaedics, 2023
- Assessing the Accuracy of Responses by the Language Model ChatGPT to Questions Regarding Bariatric SurgeryObesity Surgery, 2023
- Use of an Artificial Intelligence Conversational Agent (Chatbot) for Hip Arthroscopy Patients Following SurgeryArthroscopy, Sports Medicine, and Rehabilitation, 2023
- Patients Undergoing Primary Hip Arthroscopy Report Favorable Outcomes at Minimum 10 Year Follow‐Up: A Systematic ReviewArthroscopy, 2022
- Good 5-year outcomes after arthroscopic treatment for femoroacetabular impingement syndromeKnee Surgery, Sports Traumatology, Arthroscopy, 2019
- Hip arthroscopy: Indications, outcomes and complicationsInternational Journal of Surgery, 2018
- How Much Arthritis Is Too Much for Hip Arthroscopy: A Systematic ReviewArthroscopy, 2014
- Trends and Demographics in Hip Arthroscopy in the United StatesArthroscopy, 2013
- Are Femoral Nerve Blocks Effective for Early Postoperative Pain Management After Hip Arthroscopy?Arthroscopy, 2012
- Assessing, Controlling, and Assuring the Quality of Medical Information on the InternetJAMA, 1997