Evaluating acute image ordering for real-world patient cases via language model alignment with radiological guidelines
Open Access
- 4 August 2025
- journal article
- Published by Springer Nature in Communications Medicine
- Vol. 5 (1) , 332
Abstract
Background: Diagnostic imaging studies are increasingly important in the management of acutely presenting patients. However, ordering appropriate imaging studies in the emergency department is a challenging task with a high degree of variability among healthcare providers. To address this issue, recent work has investigated whether generative AI and large language models can be leveraged to recommend diagnostic imaging studies in accordance with evidence-based medical guidelines. However, it remains challenging to ensure that these tools can provide recommendations that correctly align with medical guidelines, especially given the limited diagnostic information available in acute care settings. Methods: In this study, we introduce a framework to intelligently leverage language models by recommending imaging studies for patient cases that align with the American College of Radiology’s Appropriateness Criteria, a set of evidence-based guidelines. To power our experiments, we introduce RadCases, a dataset of over 1500 annotated case summaries reflecting common patient presentations, and apply our framework to enable state-of-the-art language models to reason about appropriate imaging choices. Results: Using our framework, state-of-the-art language models achieve accuracy comparable to clinicians in ordering imaging studies. Furthermore, we demonstrate that our language model-based pipeline can be used as an intelligent assistant by clinicians to support image ordering workflows and improve the accuracy of acute image ordering according to the American College of Radiology’s Appropriateness Criteria. Conclusions: Our work demonstrates and validates a strategy to leverage AI-based software to improve trustworthy clinical decision-making in alignment with expert evidence-based guidelines.Keywords
All Related Versions
Funding Information
- NSF | Directorate for Computer & Information Science & Engineering | Division of Computing and Communication Foundations (1917852)
- U.S. Department of Health & Human Services | NIH | National Institute on Minority Health and Health Disparities (F30MD020264)
- Alpha Omega Alpha Honor Medical Society
- U.S. Department of Health & Human Services | NIH | National Institute of Biomedical Imaging and Bioengineering (R01EB031722)
- U.S. Department of Health & Human Services | NIH | National Center for Advancing Translational Sciences (UL1TR001878)
This publication has 88 references indexed in Scilit:
- R-SCAN: Why We Should Care!Journal of the American College of Radiology, 2016
- Curbing the Urge to ImageThe American Journal of Medicine, 2016
- Errors in imaging patients in the emergency settingBritish Journal of Radiology, 2016
- Quality Indicators for EGDAmerican Journal of Gastroenterology, 2015
- A Comparison of Diagnostic Imaging Ordering Patterns Between Advanced Practice Clinicians and Primary Care Physicians Following Office-Based Evaluation and Management VisitsJAMA Internal Medicine, 2015
- Cognitive and System Factors Contributing to Diagnostic Errors in RadiologyAmerican Journal of Roentgenology, 2013
- Evaluation of Pulmonary Embolism in the Emergency Department and Consistency With a National Quality MeasureArchives of internal medicine (1960), 2012
- Economic Evaluation in RadiologyAcademic Radiology, 2010
- Do Clinicians Use the American College of Radiology Appropriateness Criteria in the Management of Their Patients?American Journal of Roentgenology, 2009
- Online Radiology Appropriateness SurveyAcademic Radiology, 2003