Improved Fine-Tuning of In-Domain Transformer Model for Inferring COVID-19 Presence in Multi-Institutional Radiology Reports
- 2 November 2022
- journal article
- research article
- Published by Springer Nature in Journal of Digital Imaging
- Vol. 36 (1) , 164-177
Abstract
Building a document-level classifier for COVID-19 on radiology reports could help assist providers in their daily clinical routine, as well as create large numbers of labels for computer vision models. We have developed such a classifier by fine-tuning a BERT-like model initialized from RadBERT, its continuous pre-training on radiology reports that can be used on all radiology-related tasks. RadBERT outperforms all biomedical pre-trainings on this COVID-19 task (P<0.01) and helps our fine-tuned model achieve an 88.9 macro-averaged F1-score, when evaluated on both X-ray and CT reports. To build this model, we rely on a multi-institutional dataset re-sampled and enriched with concurrent lung diseases, helping the model to resist to distribution shifts. In addition, we explore a variety of fine-tuning and hyperparameter optimization techniques that accelerate fine-tuning convergence, stabilize performance, and improve accuracy, especially when data or computational resources are limited. Finally, we provide a set of visualization tools and explainability methods to better understand the performance of the model, and support its practical use in the clinical setting. Our approach offers a ready-to-use COVID-19 classifier and can be applied similarly to other radiology report classification tasks.Keywords
Funding Information
- National Institute of Biomedical Imaging and Bioengineering (75N92020C00008, 75N92020C00021)
This publication has 21 references indexed in Scilit:
- Fastai: A Layered API for Deep LearningInformation, 2020
- How to Fine-Tune BERT for Text Classification?2019
- BioBERT: a pre-trained biomedical language representation model for biomedical text miningBioinformatics, 2019
- CheXpert: A Large Chest Radiograph Dataset with Uncertainty Labels and Expert ComparisonProceedings of the AAAI Conference on Artificial Intelligence, 2019
- SciBERT: A Pretrained Language Model for Scientific Text2019
- Transfer Learning in Biomedical Natural Language Processing: An Evaluation of BERT and ELMo on Ten Benchmarking Datasets2019
- Publicly Available Clinical2019
- Bayesian Optimization2018
- Universal Language Model Fine-tuning for Text Classification2018
- Understanding noninferiority trialsKorean Journal of Pediatrics, 2012