Systematic evaluation of common natural language processing techniques to codify clinical notes
Open Access
- 7 March 2024
- journal article
- research article
- Published by Public Library of Science (PLoS) in PLOS ONE
- Vol. 19 (3) , e0298892
Abstract
Proper codification of medical diagnoses and procedures is essential for optimized health care management, quality improvement, research, and reimbursement tasks within large healthcare systems. Assignment of diagnostic or procedure codes is a tedious manual process, often prone to human error. Natural Language Processing (NLP) has been suggested to facilitate this manual codification process. Yet, little is known on best practices to utilize NLP for such applications. With Large Language Models (LLMs) becoming more ubiquitous in daily life, it is critical to remember, not every task requires that level of resource and effort. Here we comprehensively assessed the performance of common NLP techniques to predict current procedural terminology (CPT) from operative notes. CPT codes are commonly used to track surgical procedures and interventions and are the primary means for reimbursement. Our analysis of 100 most common musculoskeletal CPT codes suggest that traditional approaches can outperform more resource intensive approaches like BERT significantly (P-value = 4.4e-17) with average AUROC of 0.96 and accuracy of 0.97, in addition to providing interpretability which can be very helpful and even crucial in the clinical domain. We also proposed a complexity measure to quantify the complexity of a classification task and how this measure could influence the effect of dataset size on model’s performance. Finally, we provide preliminary evidence that NLP can help minimize the codification error, including mislabeling due to human error.Keywords
Funding Information
- Children’s Orthopaedic Surgery Foundation
- Boston Children’s Hospital Research Faculty Council
- NVIDIA Basic Research Accelerator Program
This publication has 26 references indexed in Scilit:
- Use of Natural Language Processing Algorithms to Identify Common Data Elements in Operative Notes for Knee ArthroplastyThe Journal of Arthroplasty, 2021
- Automatic multilabel detection of ICD10 codes in Dutch cardiology discharge letters using neural networksnpj Digital Medicine, 2021
- Mining of Massive Datasets2020
- SECNLP: A survey of embeddings in clinical natural language processingJournal of Biomedical Informatics, 2020
- Deep learning for time series classification: a reviewData Mining and Knowledge Discovery, 2019
- Automatic ICD code assignment of Chinese clinical notes based on multilayer attention BiRNNJournal of Biomedical Informatics, 2019
- MIMIC-III, a freely accessible critical care databaseScientific Data, 2016
- Automated Identification of Postoperative Complications Within an Electronic Medical Record Using Natural Language ProcessingJAMA, 2011
- Principal component analysisWIREs Computational Statistics, 2010
- Support-vector networksMachine Learning, 1995