Lindsay Muir

Introduction: Artificial intelligence (AI) has demonstrated transformative potential in medical education and assessment, with large language models achieving competitive results across multiple high-stakes examinations. In this study, we evaluated the performance and inter-run reliability of 10 widely adopted large language models (LLMs) on the European Board of Hand Surgery written examination. Methods: Ten LLMs were assessed on the complete 300-item European Board of Hand Surgery written examination using standardized zero-shot prompting. The models included five proprietary systems (GPT-5 Pro, Claude Sonnet 4.5, Gemini 2.5 Pro, Grok-4 and ERNIE 4.5 Turbo) and five open-source architectures (DeepSeek V3.2, Qwen3 Max, Mistral Medium 3.1, Llama 3.3 and Falcon H1). Each LLM completed five independent runs, producing 15000 answers analysed for mean accuracy, 95% confidence intervals and inter-run reliability using Cohen’s kappa ( κ ). Results: Mean accuracy across the LLMs ranged from 72 to 85%, corresponding to total European Board of Hand Surgery scores between 131 and 211 points. Seven of the 10 LLMs reached or exceeded the illustrative pass threshold of 75%, equivalent to 150 of 300 points. Proprietary systems showed consistently higher mean accuracy than open-source systems. The highest-performing LLM (GPT-5 Pro) achieved 85% accuracy with a 95% confidence interval of 84 to 86% and a mean inter-run reliability measured by Cohen’s κ of 0.739. The overall reliability across the LLMs was 0.821. Conclusions: Contemporary LLMs show robust and reproducible performance on a complex surgical certification examination, with proprietary architectures tending to outperform open-source counterparts. Although several models reached or exceeded an illustrative pass threshold, persistent gaps in subspecialty knowledge remain such as congenital anomalies and complex reconstructions. Therefore, LLMs may assist in structured learning and examination preparation but require specialist oversight and remain unsuitable for independent subspecialty decision-making. Level of evidence: Not applicable.

Background: The reliability of clinician grading of systemic sclerosis (SSc)-digital ulcers (DUs) has been reported to be poor to moderate at best, which is a major concern in clinical trial design. The aim of this study was to examine the reliability of new proposed UK Scleroderma Study Group (UKSSG) DU definitions (no ulcer, healed ulcer and DU) amongst UK SSc clinicians. A secondary aim was to examine the performance of the definitions to be used in the context of preventative studies (i.e. no ulcer vs healed ulcer/DU) and treatment studies (i.e. no ulcer/healed ulcer vs DU).

We present a novel framework for estimating the 3D poses and shapes of the carpal bones from single view fluoroscopic sequences. A hybrid statistical model representing both the pose and shape variation of the carpal bones is built, based on a number of 3D CT data sets obtained from different subjects at different poses. Given a fluoroscopic sequence, the wrist pose, carpal bone pose and bone shapes are estimated iteratively by matching the statistical model with the 2D images. A specially designed cost function enables smoothed parameter estimation across frames and constrains local bone pose with a penalty term. We have evaluated the proposed method on both simulated data and real fluoroscopic sequences and demonstrated that the relative poses of carpal bones can be accurately estimated. One condition that may be assessed using this measurement is dissociation, where the distance between the bones is larger than normal. Scaphoid-Lunate dissociation is one of the most common of these. The error of the measured 3D Scaphoid-Lunate distances were 0.75 ± 0.50 mm for simulated data (25 subjects) and 0.93 ± 0.47 mm for real data (15 subjects). We also propose a method for constructing a “standard” pathology measurement tool for automatically detecting Scaphoid-Lunate dissociation conditions, based on single-view fluoroscopic sequences. For the simulated data, it produced 100% sensitivity and specificity. For the real data, it achieved 83% sensitivity and 78% specificity.