Performance of DeepSeek-R1 and ChatGPT-5 in the Generation of North American Spine Society Clinical Guidelines for Adult Vertebral Compression Fractures: Comparative Study
Open Access
- 10 July 2026
- journal article
- research article
- Published by JMIR Publications Inc. in Journal of Medical Internet Research
- Vol. 28, e87816
Abstract
Background: Vertebral compression fractures (VCFs) impose a substantial clinical and health care burden, and their management relies on timely access to evidence-based guidelines. Large language models (LLMs) may help clinicians rapidly obtain guideline-related information, but their performance on VCF guidelines remains unclear. Objective: This study aimed to evaluate the performance of LLMs, including DeepSeek-R1 and ChatGPT-5, in generating responses consistent with VCF clinical guidelines. Methods: Using the 2024 North American Spine Society VCF clinical guidelines as the reference standard, 34 open-ended and 87 closed-ended questions were submitted to DeepSeek-R1 and ChatGPT-5. Four senior spine surgeons independently rated responses to both closed-ended and open-ended questions using a 5-point Likert scale for accuracy, consistency, self-awareness, and fabrication/falsification. For open-ended questions, comprehensiveness, clarity, and trust and confidence were additionally assessed. Subgroup analyses were performed by question type, recommendation grade, and VCF subtype, with direct comparisons between models. Results: A total of 726 responses were generated for 121 questions. For closed-ended questions, ChatGPT-5 and DeepSeek-R1 showed comparable performance in accuracy ( P =.11), self-awareness ( P =.10), and fabrication/falsification ( P =.10). DeepSeek-R1 demonstrated better consistency than ChatGPT-5 for both closed-ended and open-ended questions ( P <.001 and P =.001, respectively). For open-ended questions, the models differed significantly in comprehensiveness ( P =.03) and trust and confidence ( P =.02), but not in accuracy ( P =.42), self-awareness ( P =.22), fabrication/falsification ( P =.64), or clarity ( P =.48). Closed-ended questions generally outperformed open-ended questions. Responses to grade A-C recommendations outperformed grade I recommendations in accuracy, consistency, and fabrication/falsification (all P ≤.001) but scored lower in self-awareness ( P <.001). No significant differences were observed across VCF subtypes. Conclusions: Under a standardized clinician-oriented prompting condition, ChatGPT-5 and DeepSeek-R1 showed generally high but variable scores across evaluation dimensions, with important deficiencies remaining, particularly in interventional and surgical treatment recommendations and in questions linked to recommendation grade I. Because these findings were obtained in a controlled prompting setting, caution is warranted when extrapolating them to other query styles, clinical scenarios, or LLMs.Keywords
This publication has 46 references indexed in Scilit:
- Diagnosis and Management of Vertebral Compression Fractures.2016
- A Guideline of Selecting and Reporting Intraclass Correlation Coefficients for Reliability ResearchJournal of Chiropractic Medicine, 2016
- Management of the Elderly With Vertebral Compression FracturesNeurosurgery, 2015
- Comparison of Percutaneous Vertebroplasty and Balloon Kyphoplasty for the Treatment of Single Level Vertebral Compression Fractures: A Meta-analysis of the Literature.2015
- Osteoporosis and Its ComplicationsMedical Clinics of North America, 2014
- Vertebral compression fractures: a review of current management and multimodal therapyJournal of Multidisciplinary Healthcare, 2013
- Strengthening the reporting of observational studies in epidemiology (STROBE) statement: guidelines for reporting observational studiesBMJ, 2007
- Incidence and Economic Burden of Osteoporosis-Related Fractures in the United States, 2005–2025Journal of Bone and Mineral Research, 2007
- Impact of incident vertebral fractures on health related quality of life (HRQOL) in postmenopausal women with prevalent vertebral fracturesOsteoporosis International, 2004
- Health-Related Quality of Life in Postmenopausal Women With Low BMD With or Without Prevalent Vertebral FracturesJournal of Bone and Mineral Research, 2000