Efficiency at Scale: Investigating the Performance of Diminutive Language Models in Clinical Tasks
Authors:
Preprint
Abstract
The entry of large language models (LLMs) into research and commercial spaces has led to a trend of ever-larger models, with initial promises of generalisability, followed by a widespread desire to downsize and create specialised models without the need for complete fine-tuning, using Parameter Efficient Fine-tuning (PEFT) methods. We present an investigation into the suitability of different PEFT methods to clinical decision-making tasks, across a range of model sizes, including extremely small models with as few as $25$ million parameters. Our analysis shows that the performance of most PEFT approaches varies significantly from one task to another, with the exception of LoRA, which maintains relatively high performance across all model sizes and tasks, typically approaching or matching full fine-tuned performance. The effectiveness of PEFT methods in the clinical domain is evident, particularly for specialised models which can operate on low-cost, in-house computing infrastructure. The advantages of these models, in terms of speed and reduced training costs, dramatically outweighs any performance gain from large foundation LLMs. Furthermore, we highlight how domain-specific pre-training interacts with PEFT methods and model size, and discuss how these factors interplay to provide the best efficiency-performance trade-off. Full code available at: tbd.
Keywords
All Related Versions
This publication has 17 references indexed in Scilit:
- An Empirical Study of Catastrophic Forgetting in Large Language Models During Continual Fine-TuningIEEE Transactions on Audio, Speech and Language Processing, 2025
- Clinical Prompt Learning With Frozen Language ModelsIEEE Transactions on Neural Networks and Learning Systems, 2023
- Parameter-efficient fine-tuning of large-scale pre-trained language modelsNature Machine Intelligence, 2023
- On the effectiveness of compact biomedical transformersBioinformatics, 2023
- Clinical prediction models in psychiatry: a systematic review of two decades of progress and challengesMolecular Psychiatry, 2022
- MobileBERT: a Compact Task-Agnostic BERT for Resource-Limited Devices2020
- BioBERT: a pre-trained biomedical language representation model for biomedical text miningBioinformatics, 2019
- Automated systems for the de-identification of longitudinal clinical narratives: Overview of 2014 i2b2/UTHealth shared task Track 1Journal of Biomedical Informatics, 2015
- Evaluating temporal relations in clinical text: 2012 i2b2 ChallengeJournal of the American Medical Informatics Association, 2013
- 2010 i2b2/VA challenge on concepts, assertions, and relations in clinical textJournal of the American Medical Informatics Association, 2011