Automated ICD Coding using Extreme Multi-label Long Text Transformer-based Models
Authors:
Preprint
Abstract
Background: Encouraged by the success of pretrained Transformer models in many natural language processing tasks, their use for International Classification of Diseases (ICD) coding tasks is now actively being explored. In this study, we investigate three types of Transformer-based models, aiming to address the extreme label set and long text classification challenges that are posed by automated ICD coding tasks. Methods: The Transformer-based model PLM-ICD achieved the current state-of-the-art (SOTA) performance on the ICD coding benchmark dataset MIMIC-III. It was chosen as our baseline model to be further optimised. XR-Transformer, the new SOTA model in the general extreme multi-label text classification domain, and XR-LAT, a novel adaptation of the XR-Transformer model, were also trained on the MIMIC-III dataset. XR-LAT is a recursively trained model chain on a predefined hierarchical code tree with label-wise attention, knowledge transferring and dynamic negative sampling mechanisms. Results: Our optimised PLM-ICD model, which was trained with longer total and chunk sequence lengths, significantly outperformed the current SOTA PLM-ICD model, and achieved the highest micro-F1 score of 60.8%. The XR-Transformer model, although SOTA in the general domain, did not perform well across all metrics. The best XR-LAT based model obtained results that were competitive with the current SOTA PLM-ICD model, including improving the macro-AUC by 2.1%. Conclusion: Our optimised PLM-ICD model is the new SOTA model for automated ICD coding on the MIMIC-III dataset, while our novel XR-LAT model performs competitively with the previous SOTA PLM-ICD model.Keywords
All Related Versions
This publication has 22 references indexed in Scilit:
- A comparative study of pretrained language models for long clinical textJournal of the American Medical Informatics Association, 2022
- Hierarchical label-wise attention transformer model for explainable ICD codingJournal of Biomedical Informatics, 2022
- Domain-Specific Language Model Pretraining for Biomedical Natural Language ProcessingACM Transactions on Computing for Healthcare, 2021
- TransICD: Transformer Based Code-Wise Attention Model for Explainable ICD Coding2021
- Multi-channel, convolutional attention based neural model for automated diagnostic coding of unstructured patient discharge summariesFuture Generation Computer Systems, 2021
- Explainable automated coding of clinical notes using hierarchical label-wise attention networks and label embedding initialisationJournal of Biomedical Informatics, 2021
- Problems and Barriers during the Process of Clinical Coding: a Focus Group Study of Coders’ PerceptionsJournal of Medical Systems, 2020
- BioBERT: a pre-trained biomedical language representation model for biomedical text miningBioinformatics, 2019
- MIMIC-III, a freely accessible critical care databaseScientific Data, 2016
- A systematic literature review of automated clinical coding and classification systemsJournal of the American Medical Informatics Association, 2010