Retrospective Evaluation of an AI System to Classify Negative Musculoskeletal Trauma Radiographs

Authors:

Abstract
Study objectives: To evaluate an AI system for musculoskeletal (MSK) radiography in confidently identifying examinations without injury-related pathologies to support AI-guided discharge of patients without traumatic findings. Methods: We retrospectively sampled radiographic examinations, including one or more radiographs and a radiological report, of suspected MSK trauma from > 1,000 clinical sites in two countries. Medically trained professionals independently classified all exams. When disagreements between the classification and the original radiological report arose, adjudication was performed by a third professional. Annotators were also asked to record their confidence level of each classification. The AI system analyzed all exams and assigned them to the categories AI Positive, AI Negative , and AI Negative (very high confidence) . Performance of the AI system was assessed using error rate, false negative rate (FNR), and qualitative review of misclassified exams. Results: A total of 2,962 exams were included. The AI classified 27.6% (818/2,961; 95% CI: 26.0–29.3) of exams as AI Negative (very high confidence) . Of all exams, 0.7% (21/2,962; 95% CI: 0.0–1.0) were falsely classified as highconfidence negatives, corresponding to a false negative rate (FNR) of 2.0% (21/1,026; 95% CI: 0.0–2.9). Qualitative review of false negatives showed that the majority had no clinical consequence if correctly diagnosed during routine follow-up the following day, and no clearly high-risk exams were missed. Conclusion: The AI system identified over one-quarter of MSK trauma radiographs as confidently negative with a very low rate of false negatives, performing on par or better than the reported standard-of-care. These results suggest potential for safe, AI-driven decision support and workflow optimization for the discharge of patients with clearly negative examinations.

This publication has 14 references indexed in Scilit: