Comparison of three large language models in postoperative rehabilitation question answering after anterior cruciate ligament reconstruction based on expert ratings
Open Access
- 9 June 2026
- journal article
- Published by Springer Nature in BMC Musculoskeletal Disorders
- Vol. 27 (1)
Abstract
Large language models (LLMs) are increasingly used by patients to obtain health information. Postoperative rehabilitation after anterior cruciate ligament reconstruction (ACLR) has distinct phase boundaries and safety considerations. Therefore, responses should be not only clear and understandable, but also medically accurate, safe, and stage-fit. This study compared the performance of three publicly accessible LLMs in standardized post-ACLR rehabilitation question answering. This was a standardized, blinded, expert-rated comparative evaluation study. On a single prespecified data collection day in March 2026, 30 English-language rehabilitation questions were submitted separately to GPT-5.4, Doubao, and MiniMax-M2.7. The questions covered five postoperative rehabilitation phases. Responses were anonymized and randomly reordered before blinded rating by five orthopaedic clinicians across five domains: Accuracy, Safety, Stage-fit, Completeness, and Understandability. Paired non-parametric tests, effect size analyses, intraclass correlation coefficients, and linear mixed-effects modelling were used for statistical analysis. A total of 90 model-generated responses and 450 expert rating records were included. Overall scores differed significantly among the three models (Friedman χ² = 46.067, P < 0.001; Kendall's W = 0.768). GPT-5.4 achieved the highest overall score (4.61 ± 0.13), followed by MiniMax-M2.7 (4.53 ± 0.19), whereas Doubao had the lowest score (3.86 ± 0.29). GPT-5.4 performed best in Accuracy, Safety, and Stage-fit; MiniMax-M2.7 achieved the highest score for Completeness; and Doubao achieved the highest mean score for Understandability. Inter-rater agreement was good [ICC(3,k) = 0.893], and sensitivity analysis supported the primary findings. The three models showed distinct rating profiles in standardized single-turn post-ACLR rehabilitation question answering. Evaluation of patient-facing rehabilitation information should not rely solely on linguistic fluency, but should prioritize medical accuracy, safety, and Stage-fit. These findings provide preliminary benchmark evidence in a phase-sensitive rehabilitation setting, but they should not be interpreted as evidence supporting clinical implementation, clinician substitution, or patient benefit.Keywords
Funding Information
- National Natural Science Foundation of China (82405520)
- National Natural Science Foundation of China (82274546)
- Jiangsu Province Traditional Chinese Medicine Science and Technology Development Program (QN202322)
This publication has 39 references indexed in Scilit:
- Evaluating large language models and agents in healthcare: key challenges in clinical applicationsIntelligent Medicine, 2025
- Large Language Models for Chatbot Health Advice StudiesJAMA Network Open, 2025
- Large language models’ performances regarding common patient questions about osteoarthritis: A comparative analysis of ChatGPT-3.5, ChatGPT-4.0, and PerplexityJournal of Sport and Health Science, 2024
- ChatGPT‐3.5 and ‐4.0 Do Not Reliably Create Readable Patient Education Materials for Common Orthopaedic Upper‐ and Lower‐Extremity ConditionsArthroscopy, Sports Medicine, and Rehabilitation, 2024
- Can ChatGPT answer patient questions regarding reverse shoulder arthroplasty?Journal of ISAKOS, 2024
- Accuracy and Reliability of Chatbot Responses to Physician QuestionsJAMA Network Open, 2023
- Comparing Physician and Artificial Intelligence Chatbot Responses to Patient Questions Posted to a Public Social Media ForumJAMA Internal Medicine, 2023
- Rehabilitation After Anterior Cruciate Ligament Injury: Review of Current Literature and RecommendationsCurrent Reviews in Musculoskeletal Medicine, 2022
- What is the Evidence for and Validity of Return-to-Sport Testing after Anterior Cruciate Ligament Reconstruction Surgery? A Systematic Review and Meta-AnalysisSports Medicine, 2019
- Patient Perceptions and Current Trends in Internet Use by Orthopedic OutpatientsHSS Journal®: The Musculoskeletal Journal of Hospital for Special Surgery, 2017