Zero-shot extraction of seizure outcomes from clinical notes using generative pretrained transformers

Authors:
Preprint
Publisher website
Google Scholar
Abstract
Purpose: Pre-trained encoder transformer models have extracted information from unstructured clinic note text but require manual annotation for supervised fine-tuning. Large, Generative Pre- trained Transformers (GPTs) may streamline this process. In this study, we explore GPTs in zero- and few-shot learning scenarios to analyze clinical health records. Materials and Methods: We prompt-engineered LLAMA2 13B to optimize performance in extracting seizure freedom from epilepsy clinic notes and compared it against zero-shot and fine-tuned Bio+ClinicalBERT models. Our evaluation encompasses different prompting paradigms, including one-word answers, elaboration-based responses, prompts with date formatting instructions, and prompts with dates in context. Results: We found promising median accuracy rates in seizure freedom classification for zero-shot GPT models: one-word – 62%, elaboration – 50%, prompts with formatted dates – 62%, and prompts with dates in context – 74%. These outperform the zero-shot Clinical BERT model (25%) but fall short of the fully fine-tuned BERT model (84%). Furthermore, in sparse contexts, such as notes from general neurologists, the best performing GPT model (76%) surpasses the fine-tuned BERT model (67%) in extracting seizure freedom. Conclusion: This study demonstrates the potential of GPTs in extracting clinically relevant information from unstructured EMR text, offering insights into population trends in seizure management, drug effects, risk factors, and healthcare disparities. Moreover, GPTs exhibit superiority over task-specific models in contexts with the potential to include less precise descriptions of epilepsy and seizures, highlighting their versatility. Additionally, simple prompt engineering techniques enhance model accuracy, presenting a framework for leveraging EMR data with zero clinical annotation. Competing Interests: The authors declare no financial or non-financial competing interests.
All Related Versions

This publication has 41 references indexed in Scilit: