Generative large language models are all-purpose text analytics engines: text-to-text learning is all your need
- 17 April 2024
- journal article
- research article
- Published by Oxford University Press (OUP) in Journal of the American Medical Informatics Association
- Vol. 31 (9) , 1892-1903
Abstract
Objective: To solve major clinical natural language processing (NLP) tasks using a unified text-to-text learning architecture based on a generative large language model (LLM) via prompt tuning. Methods: We formulated 7 key clinical NLP tasks as text-to-text learning and solved them using one unified generative clinical LLM, GatorTronGPT, developed using GPT-3 architecture and trained with up to 20 billion parameters. We adopted soft prompts (ie, trainable vectors) with frozen LLM, where the LLM parameters were not updated (ie, frozen) and only the vectors of soft prompts were updated, known as prompt tuning. We added additional soft prompts as a prefix to the input layer, which were optimized during the prompt tuning. We evaluated the proposed method using 7 clinical NLP tasks and compared them with previous task-specific solutions based on Transformer models. Results and Conclusion: The proposed approach achieved state-of-the-art performance for 5 out of 7 major clinical NLP tasks using one unified generative LLM. Our approach outperformed previous task-specific transformer models by ∼3% for concept extraction and 7% for relation extraction applied to social determinants of health, 3.4% for clinical concept normalization, 3.4%-10% for clinical abbreviation disambiguation, and 5.5%-9% for natural language inference. Our approach also outperformed a previously developed prompt-based machine reading comprehension (MRC) model, GatorTron-MRC, for clinical concept and relation extraction. The proposed approach can deliver the “one model for all” promise from training to deployment using a unified generative LLM.Keywords
All Related Versions
Funding Information
- Patient-Centered Outcomes Research Institute (ME-2018C3-14754)
- National Cancer Institute (R01CA246418)
- National Institute on Aging (NIA R56AG069880, R01AG080624, R01AG083039, R01AG080991, R01AG084236, R01AG084178, R01AG076234, R33AG062884)
- Ed and Ethel Moore Alzheimer’s Disease Research Program 23A09
- National Institute of Allergy and Infectious Diseases (NIAID R01AI172875)
- National Heart, Lung, and Blood Institute (NHLBI R01HL169277)
- Cancer Informatics Shared Resource
- UF Health Cancer Center
- UF Clinical and Translational Science Institute Biomedical Informatics Program
This publication has 24 references indexed in Scilit:
- Sequence Labeling for Disambiguating Medical AbbreviationsJournal of Healthcare Informatics Research, 2023
- Overview of the 2022 n2c2 shared task on contextualized medication event extraction in clinical notesJournal of Biomedical Informatics, 2023
- Contextualized medication information extraction using Transformer-based deep learning architecturesJournal of Biomedical Informatics, 2023
- A large language model for electronic health recordsnpj Digital Medicine, 2022
- Clinical concept extraction using transformersJournal of the American Medical Informatics Association, 2020
- BioBERT: a pre-trained biomedical language representation model for biomedical text miningBioinformatics, 2019
- MIMIC-III, a freely accessible critical care databaseScientific Data, 2016
- Overview of the ShARe/CLEF eHealth Evaluation Lab 20132013
- Bidirectional LSTM Networks for Improved Phoneme Classification and Recognition2005
- Exploring semantic groups through visual approachesJournal of Biomedical Informatics, 2003