Extracting patient reported cannabis use and reasons for use from electronic health records: a benchmarking study of large language models

Authors:
Preprint
Publisher website
Google Scholar
Abstract
Key messages: What is already known on this topic Cannabis use is increasingly documented in EHR narrative text, but structured fields do not capture use status or symptom related motivations, limiting research on pain self-management strategies in autoimmune rheumatic diseases. What this study adds We developed a natural language processing pipeline using large language models to identify cannabis use status (4-class) and reasons for use (6-class), with best performance from fine-tuned GatorTron for status and an LLM for reasons. How this study might affect research, practice or policy This scalable approach can support real-world evidence studies on symptom management, medication use, and outcomes among ARD populations, and it provides a methodological template for extracting under-documented patient behaviors from narrative notes. Objective: The primary objective is to develop and evaluate a scalable and reproducible natural language processing (NLP) approach using large language models (LLM) to identify cannabis use status and reasons for cannabis use among patients with autoimmune rheumatic diseases (ARDs) from unstructured electronic health record (EHR) clinical notes. Methods and Analysis: We conducted a retrospective study using EHR clinical notes from patients with ARDs (2015–2024). Notes were screened for cannabis-related mentions using fuzzy string matching against a curated keyword lexicon with a similarity threshold of 90, extracting 50-word context windows (±25 words). Two domain experts annotated 886 randomly sampled snippets across four classes: (1) not a true cannabis mention/uncertain, (2) denial of use, (3) positive past use, and (4) positive current use. Using these annotations, we compared multiple LLM prompting strategies (zero-shot to few-shot; temperature tuning) and a fine-tuned clinical model (GatorTron 345M). For “reason for use,” 1,027 snippets were annotated into six categories: pain, nausea, sleep, anxiety/stress/mood, appetite, and not mentioned/unknown. Models were evaluated on a held-out validation set using accuracy, F1, recall, and precision. We then aggregated snippet-level predictions to patient level to describe temporal trends and subgroup differences. Results: For cannabis use status classification, the fine-tuned GatorTron model achieved the highest performance (accuracy 0.90; F1 0.91; recall 0.90; precision 0.90). For the reason of cannabis use classification, GPT-OSS-20B achieved the highest performance (accuracy 0.90; F1 0.90; recall 0.90; precision 0.92). Patient-level analyses characterized trends in documented cannabis use from 2015–2024 and compared clinical characteristics between current users and patients denying use. Conclusion: High-precision extraction of cannabis use status and reasons for use from EHR notes is feasible using a combination of fine-tuned clinical language models and LLM-based classifiers. This approach enables scalable measurement of patient-reported symptom self-management strategies in ARDs, supporting observational research and potential clinical decision support.
Funding Information
  • NIDA (T32DA035165)
  • National Institute of Arthritis and Musculoskeletal and Skin Diseases (K01AR079039)

This publication has 23 references indexed in Scilit: