Use of Natural Language Processing to Infer Sites of Metastatic Disease From Radiology Reports at Scale
Open Access
- 1 May 2024
- journal article
- research article
- Published by American Society of Clinical Oncology (ASCO) in JCO Clinical Cancer Informatics
- Vol. 8 (8) , e2300122
Abstract
PURPOSE: To evaluate natural language processing (NLP) methods to infer metastatic sites from radiology reports. METHODS: A set of 4,522 computed tomography (CT) reports of 550 patients with 14 types of cancer was used to fine-tune four clinical large language models (LLMs) for multilabel classification of metastatic sites. We also developed an NLP information extraction (IE) system (on the basis of named entity recognition, assertion status detection, and relation extraction) for comparison. Model performances were measured by F1 scores on test and three external validation sets. The best model was used to facilitate analysis of metastatic frequencies in a cohort study of 6,555 patients with 53,838 CT reports. RESULTS: The RadBERT, BioBERT, GatorTron-base, and GatorTron-medium LLMs achieved F1 scores of 0.84, 0.87, 0.89, and 0.91, respectively, on the test set. The IE system performed best, achieving an F1 score of 0.93. F1 scores of the IE system by individual cancer type ranged from 0.89 to 0.96. The IE system attained F1 scores of 0.89, 0.83, and 0.81, respectively, on external validation sets including additional cancer types, positron emission tomography-CT ,and magnetic resonance imaging scans, respectively. In our cohort study, we found that for colorectal cancer, liver-only metastases were higher in de novo stage IV versus recurrent patients (29.7% v 12.2%; P < .001). Conversely, lung-only metastases were more frequent in recurrent versus de novo stage IV patients (17.2% v 7.3%; P < .001). CONCLUSION: We developed an IE system that accurately infers metastatic sites in multiple primary cancers from radiology reports. It has explainable methods and performs better than some clinical LLMs. The inferred metastatic phenotypes could enhance cancer research databases and clinical trial matching, and identify potential patients for oligometastatic interventions.Keywords
This publication has 30 references indexed in Scilit:
- A large language model for electronic health recordsnpj Digital Medicine, 2022
- Spark NLP: Natural Language Understanding at ScaleSoftware Impacts, 2021
- Molecular principles of metastasis: a hallmark of cancer revisitedSignal Transduction and Targeted Therapy, 2020
- Assessment of Deep Natural Language Processing in Ascertaining Oncologic Outcomes From Radiology ReportsJAMA Oncology, 2019
- BioBERT: a pre-trained biomedical language representation model for biomedical text miningBioinformatics, 2019
- Are 90% of deaths from cancer caused by metastases?Cancer Medicine, 2019
- Metastasis Organotropism: Redefining the Congenial SoilDevelopmental Cell, 2019
- Automatic Normalization of Anatomical Phrases in Radiology Reports Using Unsupervised LearningJournal of Digital Imaging, 2018
- Survival and prognostic factors in 321 patients treated with stereotactic body radiotherapy for oligo-metastasesRadiotherapy and Oncology, 2015
- Surgical treatment of oligometastatic non-small cell lung cancerLung Cancer, 2010