Evaluating approaches of training a generative large language model for multi-label classification of unstructured electronic health records
Preprint
Abstract
Multi-label classification of unstructured electronic health records (EHR) is challenging due to the semantic complexity of textual data. Identifying the most effective machine learning method for EHR classification is useful in real-world clinical settings. Advances in natural language processing (NLP) using large language models (LLMs) offer promising solutions. Therefore, this experimental research aims to test the effects of zero-shot and few-shot learning prompting, with and without parameter-efficient fine-tuning (PEFT) and retrieval-augmented generation (RAG) of LLMs, on the multi-label classification of unstructured EHR data from residential aged care facilities (RACFs) in Australia. The four clinical tasks examined are agitation in dementia, depression in dementia, frailty index, and malnutrition risk factors, using the Llama 3.1-8B. Performance evaluation includes accuracy, macro-averaged precision, recall, and F1 score, supported by non-parametric statistical analyses. Results indicate that both zero-shot and few-shot learning, regardless of the use of PEFT and RAG, demonstrate equivalent performance across the clinical tasks when using the same prompting template. Few-shot learning consistently outperforms zero-shot learning when neither PEFT nor RAG is applied. Notably, PEFT significantly enhances model performance in both zero-shot and few-shot learning; however, RAG improves performance only in few-shot learning. After PEFT, the performance of zero-shot learning is equal to that of few-shot learning across clinical tasks. Additionally, few-shot learning with RAG surpasses zero-shot learning with RAG, while no significant difference exists between few-shot learning with RAG and zero-shot learning with PEFT. These findings offer crucial insights into LLMs for researchers, practitioners, and stakeholders utilizing LLMs in clinical document analysis.Keywords
This publication has 38 references indexed in Scilit:
- Applying generative AI with retrieval augmented generation to summarize and extract key clinical information from electronic health recordsJournal of Biomedical Informatics, 2024
- The shaky foundations of large language models and foundation models for electronic health recordsnpj Digital Medicine, 2023
- Cardiovascular Care Innovation through Data-Driven Discoveries in the Electronic Health RecordThe American Journal of Cardiology, 2023
- Large language models encode clinical knowledgeNature, 2023
- Health system-scale language models are all-purpose prediction enginesNature, 2023
- Comparison of text preprocessing methodsNatural Language Engineering, 2022
- Ensemble application of bidirectional LSTM and GRU for aspect category detection with imbalanced dataNeural Computing and Applications, 2021
- Health Quest: A generalized clinical decision support system with multi-label classificationJournal of King Saud University Computer and Information Sciences, 2021
- Leveraging Big Data to Model the Likelihood of Developing Psychological Conditions After a ConcussionProcedia Computer Science, 2015
- Shared decision making in clinical medicine: past research and future directionsAmerican Journal of Preventive Medicine, 1999