Generative Large Language Models for Detection of Speech Recognition Errors in Radiology Reports
Top Cited Papers
- 1 March 2024
- journal article
- research article
- Published by Radiological Society of North America (RSNA) in Radiology: Artificial Intelligence
- Vol. 6 (2) , e230205
Abstract
"Just Accepted" papers have undergone full peer review and have been accepted for publication in Radiology: Artificial Intelligence. This article will undergo copyediting, layout, and proof review before it is published in its final version. Please note that during production of the final copyedited article, errors may be discovered which could affect the content. This study evaluated the ability of generative large language models (LLMs) to detect speech recognition errors in radiology reports. A dataset of 3,233 CT and MRI reports was assessed by radiologists for speech recognition errors. Errors were categorized as clinically significant or not clinically significant. Performances of five generative LLMs-GPT-3.5-turbo, GPT-4, text-davinci-003, Llama-v2-70B-chat, and Bard-were compared in detecting these errors, using manual error detection as the reference standard. Prompt engineering was used to optimize model performance. GPT-4 demonstrated high accuracy in detecting clinically significant errors (precision 76.9%, recall 100%, F1 86.9%) and not clinically significant errors (93.9% precision, 94.7% recall, 94.3% F1). Text-davinci-003 achieved F1 scores of 72% and 46.6% for clinically significant and not clinically significant errors, respectively. GPT-3.5-turbo obtained 59.1% and 32.2% F1 scores, while Llama-v2-70B-chat scored 72.8% and 47.7%. Bard showed the lowest accuracy, with F1 scores of 47.5% and 20.9%. GPT-4 effectively identified challenging errors of nonsense phrases and internally inconsistent statements. Longer reports, resident dictation, and overnight shifts were associated with higher error rates. In conclusion, advanced generative LLMs show potential for automatic detection of speech recognition errors in radiology reports. ©RSNA, 2024.Keywords
This publication has 16 references indexed in Scilit:
- Evaluating progress in automatic chest X-ray radiology report generationPatterns, 2023
- Evaluating GPT as an Adjunct for Radiologic Decision Making: GPT-4 Versus GPT-3.5 in a Breast Imaging PilotJournal of the American College of Radiology, 2023
- Leveraging GPT-4 for Post Hoc Transformation of Free-text Radiology Reports into Structured Reporting: A Multilingual Feasibility StudyRadiology, 2023
- Automated deidentification of radiology reports combining transformer and “hide in plain sight” rule-based methodsJournal of the American Medical Informatics Association, 2022
- Trainee radiologist reports as a source of confirmation bias in radiologyClinical Radiology, 2018
- Structured Reporting in RadiologyAcademic Radiology, 2018
- Improving Radiology Report Quality by Rapidly Notifying Radiologist of Report ErrorsJournal of Digital Imaging, 2015
- Creation and Implementation of Department-Wide Structured Reports: An Analysis of the Impact on Error Rate in Radiology ReportsJournal of Digital Imaging, 2014
- Non-Clinical Errors Using Voice Recognition Dictation Software for Radiology Reports: A Retrospective AuditJournal of Digital Imaging, 2010
- Voice Recognition Dictation: Radiologist as TranscriptionistJournal of Digital Imaging, 2007