Clinical Prompt Learning With Frozen Language Models
- 11 August 2023
- journal article
- research article
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Transactions on Neural Networks and Learning Systems
- Vol. 35 (11) , 16453-16463
Abstract
When the first transformer-based language models were published in the late 2010s, pretraining with general text and then fine-tuning the model on a task-specific dataset often achieved the state-of-the-art performance. However, more recent work suggests that for some tasks, directly prompting the pretrained model matches or surpasses fine-tuning in performance with few or no model parameter updates required. The use of prompts with language models for natural language processing (NLP) tasks is known as prompt learning. We investigated the viability of prompt learning on clinically meaningful decision tasks and directly compared this with more traditional fine-tuning methods. Results show that prompt learning methods were able to match or surpass the performance of traditional fine-tuning with up to 1000 times fewer trainable parameters, less training time, less training data, and lower computation resource requirements. We argue that these characteristics make prompt learning a very desirable alternative to traditional fine-tuning for clinical tasks, where the computational resources of public health providers are limited, and where data can often not be made available or not be used for fine-tuning due to patient privacy concerns. The complementary code to reproduce the experiments presented in this work can be found at https://github.com/NtaylorOX/Public_Clinical_Prompt .Keywords
All Related Versions
Funding Information
- Engineering & Physical Sciences Research Council (EPSRC) Center for Doctoral Training in Health Data Science (EP/S02428X/1)
- National Institute for Health and Care Research “AI for Health and Social Care Award” (NIHR-AI-AWARD0-2183)
- GlaxoSmithKline
- Rosetrees (A2926)
This publication has 9 references indexed in Scilit:
- Pre-train, Prompt, and Predict: A Systematic Survey of Prompting Methods in Natural Language ProcessingACM Computing Surveys, 2023
- Code Synonyms Do Matter: Multiple Synonyms Matching Network for Automatic ICD Coding2022
- Domain-Specific Language Model Pretraining for Biomedical Natural Language ProcessingACM Transactions on Computing for Healthcare, 2021
- Clinical Outcome Prediction from Admission Notes using Self-Supervised Knowledge Integration2021
- Prompting Contrastive Explanations for Commonsense Reasoning Tasks2021
- Real-world effectiveness, its predictors and onset of action of cholinesterase inhibitors and memantine in dementia: retrospective health record studyThe British Journal of Psychiatry, 2020
- Identifying Predictors of Suicide in Severe Mental Illness: A Feasibility Study of a Clinical Prediction Rule (Oxford Mental Illness and Suicide Tool or OxMIS)Frontiers in Psychiatry, 2020
- MIMIC-Extract2020
- MIMIC-III, a freely accessible critical care databaseScientific Data, 2016