Retrospective Evaluation of an AI System to Classify Negative Musculoskeletal Trauma Radiographs
Preprint
Abstract
Study objectives: To evaluate an AI system for musculoskeletal (MSK) radiography in confidently identifying examinations without injury-related pathologies to support AI-guided discharge of patients without traumatic findings. Methods: We retrospectively sampled radiographic examinations, including one or more radiographs and a radiological report, of suspected MSK trauma from > 1,000 clinical sites in two countries. Medically trained professionals independently classified all exams. When disagreements between the classification and the original radiological report arose, adjudication was performed by a third professional. Annotators were also asked to record their confidence level of each classification. The AI system analyzed all exams and assigned them to the categories AI Positive, AI Negative , and AI Negative (very high confidence) . Performance of the AI system was assessed using error rate, false negative rate (FNR), and qualitative review of misclassified exams. Results: A total of 2,962 exams were included. The AI classified 27.6% (818/2,961; 95% CI: 26.0–29.3) of exams as AI Negative (very high confidence) . Of all exams, 0.7% (21/2,962; 95% CI: 0.0–1.0) were falsely classified as highconfidence negatives, corresponding to a false negative rate (FNR) of 2.0% (21/1,026; 95% CI: 0.0–2.9). Qualitative review of false negatives showed that the majority had no clinical consequence if correctly diagnosed during routine follow-up the following day, and no clearly high-risk exams were missed. Conclusion: The AI system identified over one-quarter of MSK trauma radiographs as confidently negative with a very low rate of false negatives, performing on par or better than the reported standard-of-care. These results suggest potential for safe, AI-driven decision support and workflow optimization for the discharge of patients with clearly negative examinations.Keywords
This publication has 14 references indexed in Scilit:
- Evaluation of commercial AI algorithms for the detection of fractures, effusions, and dislocations on real-world clinical data: A prospective registry studyRadiography, 2025
- Application of artificial intelligence in the diagnosis of scaphoid fractures: impact of automated detection of scaphoid fractures in a real-life studyLa radiologia medica, 2025
- Comparative accuracy of two commercial AI algorithms for musculoskeletal trauma detection in emergency radiographsEmergency Radiology, 2025
- Real-life benefit of artificial intelligence-based fracture detection in a pediatric emergency departmentEuropean Radiology, 2025
- Accuracy of radiologists and radiology residents in detection of paediatric appendicular fractures with and without artificial intelligenceBMJ Health & Care Informatics, 2024
- External validation of an artificial intelligence solution for the detection of elbow fractures and joint effusions in childrenDiagnostic and Interventional Imaging, 2023
- Comparison of diagnostic performance of a deep learning algorithm, emergency physicians, junior radiologists and senior radiologists in the detection of appendicular fractures in childrenPediatric Radiology, 2023
- Assessment of an AI Aid in Detection of Adult Appendicular Skeletal Fractures by Emergency Physicians and Radiologists: A Multicenter Cross-sectional Diagnostic StudyRadiology, 2021
- Assessment of a deep-learning system for fracture detection in musculoskeletal radiographsnpj Digital Medicine, 2020
- Fracture and Dislocation Classification Compendium—2018Journal of Orthopaedic Trauma, 2018