Training a Deep Contextualized Language Model for International Classification of Diseases, 10th Revision Classification via Federated Learning: Model Development and Validation Study
Authors:
- Pei-Fu ChenPCPei-Fu Chen
- Tai-Liang HeTHTai-Liang He
- Sheng-Che LinSLSheng-Che Lin
- Yuan-Chia ChuYCYuan-Chia Chu
- Chen-Tsung KuoCKChen-Tsung Kuo
- Feipei LaiFLFeipei Lai
- Ssu-Ming WangSWSsu-Ming Wang
- Wan-Xuan ZhuWZWan-Xuan Zhu
- Kuan-Chih ChenKCKuan-Chih Chen
- Lu-Cheng KuoLKLu-Cheng Kuo
- Fang-Ming HungFHFang-Ming Hung
- Yu-Cheng LinYLYu-Cheng Lin
Open Access
- 10 November 2022
- journal article
- research article
- Published by JMIR Publications Inc. in JMIR Medical Informatics
- Vol. 10 (11) , e41342
Abstract
Background: The automatic coding of clinical text documents by using the International Classification of Diseases, 10th Revision (ICD-10) can be performed for statistical analyses and reimbursements. With the development of natural language processing models, new transformer architectures with attention mechanisms have outperformed previous models. Although multicenter training may increase a model’s performance and external validity, the privacy of clinical documents should be protected. We used federated learning to train a model with multicenter data, without sharing data per se. Objective: This study aims to train a classification model via federated learning for ICD-10 multilabel classification. Methods: Text data from discharge notes in electronic medical records were collected from the following three medical centers: Far Eastern Memorial Hospital, National Taiwan University Hospital, and Taipei Veterans General Hospital. After comparing the performance of different variants of bidirectional encoder representations from transformers (BERT), PubMedBERT was chosen for the word embeddings. With regard to preprocessing, the nonalphanumeric characters were retained because the model’s performance decreased after the removal of these characters. To explain the outputs of our model, we added a label attention mechanism to the model architecture. The model was trained with data from each of the three hospitals separately and via federated learning. The models trained via federated learning and the models trained with local data were compared on a testing set that was composed of data from the three hospitals. The micro F1 score was used to evaluate model performance across all 3 centers. Results: The F1 scores of PubMedBERT, RoBERTa (Robustly Optimized BERT Pretraining Approach), ClinicalBERT, and BioBERT (BERT for Biomedical Text Mining) were 0.735, 0.692, 0.711, and 0.721, respectively. The F1 score of the model that retained nonalphanumeric characters was 0.8120, whereas the F1 score after removing these characters was 0.7875—a decrease of 0.0245 (3.11%). The F1 scores on the testing set were 0.6142, 0.4472, 0.5353, and 0.2522 for the federated learning, Far Eastern Memorial Hospital, National Taiwan University Hospital, and Taipei Veterans General Hospital models, respectively. The explainable predictions were displayed with highlighted input words via the label attention architecture. Conclusions: Federated learning was used to train the ICD-10 classification model on multicenter clinical text while protecting data privacy. The model’s performance was better than that of models that were trained locally.Keywords
This publication has 17 references indexed in Scilit:
- TransICD: Transformer Based Code-Wise Attention Model for Explainable ICD Coding2021
- Automatic multilabel detection of ICD10 codes in Dutch cardiology discharge letters using neural networksnpj Digital Medicine, 2021
- A review of applications in federated learningComputers & Industrial Engineering, 2020
- A narrative review of the impact of the transition to ICD-10 and ICD-10-CM/PCSJAMIA Open, 2019
- BioBERT: a pre-trained biomedical language representation model for biomedical text miningBioinformatics, 2019
- Federated Learning in Distributed Medical Databases: Meta-Analysis of Large-Scale Subcortical Brain Data2019
- Two-stage Federated Phenotyping and Patient Representation Learning2019
- Emotions extraction from Arabic tweetsInternational Journal of Computers and Applications, 2018
- Explainable Prediction of Medical Codes from Clinical Text2018
- Impact of the Transition to ICD-10 on Medicare Inpatient Hospital PaymentsMedicare & Medicaid Research Review, 2011