Responses From ChatGPT‐4 Show Limited Correlation With Expert Consensus Statement on Anterior Shoulder Instability
Open Access
- 5 March 2024
- journal article
- Published by Wiley in Arthroscopy, Sports Medicine, and Rehabilitation
- Vol. 6 (3) , 100923
Abstract
Purpose: To compare the similarity of answers provided by Generative Pretrained Transformer‐4 (GPT‐4) with those of a consensus statement on diagnosis, nonoperative management, and Bankart repair in anterior shoulder instability (ASI). Methods: An expert consensus statement on ASI published by Hurley et al. in 2022 was reviewed and questions laid out to the expert panel were extracted. GPT‐4, the subscription version of ChatGPT, was queried using the same set of questions. Answers provided by GPT‐4 were compared with those of the expert panel and subjectively rated for similarity by 2 experienced shoulder surgeons. GPT‐4 was then used to rate the similarity of its own responses to the consensus statement, classifying them as low, medium, or high. Rates of similarity as classified by the shoulder surgeons and GPT‐4 were then compared and interobserver reliability calculated using weighted κ scores. Results: The degree of similarity between responses of GPT‐4 and the ASI consensus statement, as defined by shoulder surgeons, was high in 25.8%, medium in 45.2%, and low 29% of questions. GPT‐4 assessed similarity as high in 48.3%, medium in 41.9%, and low 9.7% of questions. Surgeons and GPT‐4 reached consensus on the classification of 18 questions (58.1%) and disagreement on 13 questions (41.9%). Conclusions: The responses generated by artificial intelligence exhibit limited correlation with an expert statement on the diagnosis and treatment of ASI. Clinical Relevance: As the use of artificial intelligence becomes more prevalent, it is important to understand how closely information resembles content produced by human authors.Keywords
This publication has 12 references indexed in Scilit:
- Exploring the potential of ChatGPT as a supplementary tool for providing orthopaedic informationKnee Surgery, Sports Traumatology, Arthroscopy, 2023
- Assessing ChatGPT Responses to Common Patient Questions Regarding Total Hip ArthroplastyJournal of Bone and Joint Surgery, 2023
- Can Artificial Intelligence Pass the American Board of Orthopaedic Surgery Examination? Orthopaedic Residents Versus ChatGPTClinical Orthopaedics and Related Research, 2023
- ChatGPT Utility in Healthcare Education, Research, and Practice: Systematic Review on the Promising Perspectives and Valid ConcernsHealthcare, 2023
- Artificial intelligence and ChatGPT in Orthopaedics and sports medicineJournal of Experimental Orthopaedics, 2023
- Comprehensive Review of Shoulder Instability Includes Diagnosis, Nonoperative Management, Bankart, Latarjet, Remplissage, Glenoid Bone‐Grafting, Revision Surgery, Rehabilitation and Return to Play, and Clinical Follow‐UpArthroscopy, 2022
- Anterior Shoulder Instability Part I—Diagnosis, Nonoperative Management, and Bankart Repair—An International Consensus StatementArthroscopy, 2021
- Diagnosis and Management of Traumatic Anterior Shoulder InstabilityJournal of the American Academy of Orthopaedic Surgeons, 2020
- The Bankart repair: past, present, and futureJournal of Shoulder and Elbow Surgery, 2020
- Position and Duration of Immobilization After Primary Anterior Shoulder DislocationJournal of Bone and Joint Surgery, 2010