A comparative study of large language model-based zero-shot inference and task-specific supervised classification of breast cancer pathology reports
Open Access
- 20 June 2024
- journal article
- research article
- Published by Oxford University Press (OUP) in Journal of the American Medical Informatics Association
- Vol. 31 (10) , 2315-2327
Abstract
Objective: Although supervised machine learning is popular for information extraction from clinical notes, creating large annotated datasets requires extensive domain expertise and is time-consuming. Meanwhile, large language models (LLMs) have demonstrated promising transfer learning capability. In this study, we explored whether recent LLMs could reduce the need for large-scale data annotations. Materials and Methods: We curated a dataset of 769 breast cancer pathology reports, manually labeled with 12 categories, to compare zero-shot classification capability of the following LLMs: GPT-4, GPT-3.5, Starling, and ClinicalCamel, with task-specific supervised classification performance of 3 models: random forests, long short-term memory networks with attention (LSTM-Att), and the UCSF-BERT model. Results: Across all 12 tasks, the GPT-4 model performed either significantly better than or as well as the best supervised model, LSTM-Att (average macro F1-score of 0.86 vs 0.75), with advantage on tasks with high label imbalance. Other LLMs demonstrated poor performance. Frequent GPT-4 error categories included incorrect inferences from multiple samples and from history, and complex task design, and several LSTM-Att errors were related to poor generalization to the test set. Discussion: On tasks where large annotated datasets cannot be easily collected, LLMs can reduce the burden of data labeling. However, if the use of LLMs is prohibitive, the use of simpler models with large annotated datasets can provide comparable results. Conclusions: GPT-4 demonstrated the potential to speed up the execution of clinical NLP studies by reducing the need for large annotated datasets. This may increase the utilization of NLP-based variables and outcomes in clinical studies.Keywords
Funding Information
- National Cancer Institute
- National Institutes of Health (P30CA082103)
- FDA (U01FD005978)
- UCSF–Stanford Center of Excellence in Regulatory Sciences and Innovation
- NIH (UL1 TR001872)
This publication has 29 references indexed in Scilit:
- Artificial Intelligence–Generated Draft Replies to Patient Inbox MessagesJAMA Network Open, 2024
- Generative Artificial Intelligence to Transform Inpatient Discharge Summaries to Patient-Friendly Language and FormatJAMA Network Open, 2024
- Diagnostic Accuracy of a Large Language Model in Pediatric Case StudiesJAMA Pediatrics, 2024
- A comprehensive evaluation of large Language models on benchmark biomedical text processing tasksComputers in Biology and Medicine, 2024
- Extracting structured information from unstructured histopathology reports using generative pre‐trained transformer 4 ( GPT ‐4)The Journal of Pathology, 2023
- Zero-shot interpretable phenotyping of postpartum hemorrhage using large language modelsnpj Digital Medicine, 2023
- Comparative performance of humans versus GPT-4.0 and GPT-3.5 in the self-assessment program of American Academy of OphthalmologyScientific Reports, 2023
- Comparing ChatGPT and GPT-4 performance in USMLE soft skill assessmentsScientific Reports, 2023
- A survey on clinical natural language processing in the United Kingdom from 2007 to 2022npj Digital Medicine, 2022
- Large Scale Semi-Automated Labeling of Routine Free-Text Clinical Records for Deep LearningJournal of Digital Imaging, 2018