Parameter-Efficient Fine-Tuning of LLaMA for the Clinical Domain
Preprint
Abstract
Adapting pretrained language models to novel domains, such as clinical applications, traditionally involves retraining their entire set of parameters. However, this approach is increasingly proven to be impractical owing to the substantial computational requirements associated with training such large language models. To address this issue, Parameter-Efficient Fine-Tuning (PEFT) techniques offer a viable solution by selectively fine-tuning a small subset of additional parameters, significantly reducing the computational requirements for domain adaptation. In this study, we propose Clinical LLaMA-LoRA, a PEFT adapter layer built upon the open-sourced LLaMA model. Clinical LLaMA-LoRA is trained using clinical notes obtained from the MIMIC-IV database, thereby creating a specialised adapter designed for the clinical domain. Additionally, we propose a two-step PEFT framework which fuses Clinical LLaMA-LoRA with Downstream LLaMA-LoRA, another PEFT adapter specialised for downstream tasks. We evaluate this framework on multiple clinical outcome prediction datasets, comparing it to clinically trained language models. Our proposed framework achieves a state-of-the-art AUROC score averaged across all clinical downstream tasks. We observe substantial improvements of 6-9% AUROC score in the large-scale multilabel classification tasks, such as diagnoses and procedures classification.Keywords
All Related Versions
This publication has 17 references indexed in Scilit:
- Toward expert-level medical question answering with large language modelsNature Medicine, 2025
- Large language models encode clinical knowledgeNature, 2023
- MIMIC-IV, a freely accessible electronic health record datasetScientific Data, 2023
- A large language model for electronic health recordsnpj Digital Medicine, 2022
- BioGPT: generative pre-trained transformer for biomedical text generation and miningBriefings in Bioinformatics, 2022
- Domain-Specific Language Model Pretraining for Biomedical Natural Language ProcessingACM Transactions on Computing for Healthcare, 2021
- Prefix-Tuning: Optimizing Continuous Prompts for Generation2021
- Clinical XLNet: Modeling Sequential Clinical Notes and Predicting Prolonged Mechanical Ventilation2020
- BioBERT: a pre-trained biomedical language representation model for biomedical text miningBioinformatics, 2019
- MIMIC-III, a freely accessible critical care databaseScientific Data, 2016