Performance and reliability of state-of-the-art LLMs in complex hand surgery scenarios: A prospective cross-sectional, double-blinded study
Author:
Open Access
- 9 January 2026
- journal article
- Published by SAGE Publications in Journal of Orthopaedic Surgery
- Vol. 34 (1)
Abstract
Background: Integrating large language models (LLMs) into decision-making and education has shown promise across various healthcare disciplines. The study aimed to evaluate the performance of leading LLMs—ChatGPT-5, Gemini 2, Grok 3, and DeepSeek R1—in accurately responding to structured multiple-choice and open-ended queries about complex case scenarios in hand surgery. Methods: A prospective cross-sectional analysis used 50 clinically relevant, guideline-based case scenarios developed for hand surgery. Each scenario consisted of four open-ended and two multiple-choice questions, totaling 300 points per LLM. Responses were independently assessed by blinded expert reviewers using a standardized six-point Likert scale evaluating accuracy, completeness, and adherence to international surgical guidelines. Results: In multiple-choice queries, Gemini (5.9 ± 0.2) and Grok (5.9 ± 0.1) outperformed ChatGPT (5.7 ± 0.3; p = 0.031 and p = 0.009, respectively) and DeepSeek (5.6 ± 0.4; p = 0.004 and p = 0.001, respectively). In open-ended queries, Gemini (5.6 ± 0.3 accuracy) and Grok (5.5 ± 0.4 accuracy) demonstrated superior results across all measured dimensions—accuracy, completeness, and guideline adherence—markedly surpassing ChatGPT (5.1 ± 0.5 accuracy, p < 0.001) and DeepSeek (4.9 ± 0.6 accuracy; p < 0.001). Notably, Gemini and Grok demonstrated consistently high performance with minimal variability, while ChatGPT, particularly DeepSeek, exhibited considerable inconsistency in complex clinical judgments. Conclusion: Gemini 2 and Grok 3 showed reliable and clinically relevant performance, positioning them as promising adjunctive tools for decision-making and education in hand surgery. The limitations in ChatGPT-5 and the significant shortcomings of DeepSeek underscore the necessity for cautious deployment and continued refinement.Keywords
This publication has 22 references indexed in Scilit:
- Assessment of Large Language Models (LLMs) in decision-making support for gynecologic oncologyComputational and Structural Biotechnology Journal, 2024
- Leveraging ChatGPT to Produce Patient Education Materials for Common Hand ConditionsJournal of Hand Surgery Global Online, 2024
- Evaluating the Effectiveness of advanced large language models in medical Knowledge: A Comparative study using Japanese national medical examinationInternational Journal of Medical Informatics, 2024
- ChatGPT-4 Surpasses Residents: A Study of Artificial Intelligence Competency in Plastic Surgery In-service Examinations and Its Advancements from ChatGPT-3.5Plastic and Reconstructive Surgery - Global Open, 2024
- Radiologic Decision-Making for Imaging in Pulmonary Embolism: Accuracy and Reliability of Large Language Models—Bing, Claude, ChatGPT, and PerplexityIndian Journal of Radiology and Imaging, 2024
- Can AI Answer My Questions? Utilizing Artificial Intelligence in the Perioperative Assessment for Abdominoplasty PatientsAesthetic Plastic Surgery, 2024
- Performance of ChatGPT on American Board of Surgery In-Training Examination Preparation QuestionsJournal of Surgical Research, 2024
- ChatGPT Earns American Board Certification in Hand SurgeryHand Surgery and Rehabilitation, 2024
- Response Generated by Large Language Models Depends on the Structure of the PromptIndian Journal of Radiology and Imaging, 2024
- Leveraging Large Language Models (LLM) for the Plastic Surgery Resident Training: Do They Have a Role?Indian Journal of Plastic Surgery, 2023