Travis Zack

Background: FOLFIRINOX is a cornerstone regimen for eligible patients with pancreatic ductal adenocarcinoma (PDAC), but its clinical benefit is limited by substantial toxicity and frequent dose modification. In real-world practice, dose modifications are often individualized, and the clinical factors associated with these decisions remain incompletely characterized. Objective: To develop and evaluate an electronic medical record (EMR)-based machine-learning framework for modeling cycle-specific FOLFIRINOX dose modification decisions in patients with PDAC. Methods: We included patients with PDAC who received FOLFIRINOX at UCSF oncology clinics between November 2011 and December 2023. Predictors included demographic, clinical, laboratory, and treatment variables derived from the EMR. Logistic regression, random forest, and XGBoost models were trained using group-based 5-fold cross-validation to predict cycle-specific dose modifications for 5-fluorouracil, irinotecan, and oxaliplatin. Model performance was evaluated using area under the receiver operating characteristic curve. Results: The cohort included 514 patients receiving FOLFIRINOX across 5,041 treatment cycles. The mean age was 59 years, 60% of patients were White, 41% had a history of smoking, and patients received a median of 6 chemotherapy cycles. More than 60% of patients required at least one dose modification during treatment. XGBoost demonstrated the highest performance across component drugs, with AUCs ranging from 0.53 to 0.70. Clinically plausible predictors of irinotecan and oxaliplatin dose modification included hepatic and renal function markers, cumulative drug exposure, treatment-related symptoms, and demographic or behavioral characteristics. Conclusion: We developed an EMR-based machine-learning framework to model real-world FOLFIRINOX dose modification and identified clinically plausible, routinely available predictors, particularly for irinotecan and oxaliplatin. Variable model performance suggests that dosing decisions are only partially captured by structured EMR data, highlighting both the limitations of current data-driven approaches and clinical domains where ML-based models may support individualized dosing and toxicity surveillance. Future informatics efforts should incorporate dose-modification rationale, patient-reported and functional outcomes, and validation across diverse practice settings.

PURPOSE: Biomarkers, or specific somatic alterations, are increasingly required for clinical trial eligibility. Finding and enrolling patients with these biomarkers is essential not only for continuous progress in the treatment of disease but also for democratizing clinical trial participation. Here, we use data from the National Cancer Institute Clinical Trials Reporting Program (NCI CTRP), combined with large language model applications, to survey the current landscape of cancer clinical trials. METHODS: We extracted 20,894 trials from Cancer.gov from the application programming interface (API) of the NCI CTRP. We quantified biomarker rates in cancer subtypes, described the geographic distribution of trial sites, and identified failure causes for these trials. Finally, we built an application from this API to match patients with clinical trials. RESULTS: We showed that 5,044 of the 20,894 interventional clinical trials contained biomarker eligibility data and trials tended to cluster around large academic centers and cities. We identified 630 biomarkers in 36 cancer subtypes and show that most biomarkers are used as eligibility criteria for multiple cancer subtypes. We highlight that the difficulties with accrual and sponsorship were the most common reason for discontinuing clinical trials. Finally, we demonstrate a novel method to automatically match natural language queries with eligible clinical trials, NCI Clinical Trials Navigator. CONCLUSION: A survey of our clinical genomics showed that many individuals likely have mutations that would make them eligible for biomarker-driven trials. We used the NCI Clinical Trials database to show that the distribution of biomarker trials across the United States limits access for many patients and likely leads to the frequent trial termination because of inadequate accrual. Finally, we built an automated publicly available tool that can improve patient-to-trial biomarker-based matching.

PURPOSE: The increasing use of patient portal messages has enhanced patient-provider communication. However, the high volume of these messages has also contributed to physician burnout. METHODS: Patient-generated portal messages sent to a single cancer center from 2011 to 2023 were extracted. BERTopic, a natural language processing topic modeling technique based on large language models, was optimized. For further categorization, the topic words were labeled using GPT-4, followed by review by two oncologists. Uniform Manifold Approximation and Projection was used for dimensionality reduction and visualizing topics. Message volume changes over time were assessed using a Student's t test. RESULTS: A total of 2,280,851 messages were analyzed. The monthly average number of messages increased from 2,071 in 2012 to 43,430 in 2022 ( P < .001). There was a significant rise in message volume after the COVID-19 pandemic, with a posterior probability of a causal effect of 96.4% ( P = .04). Scheduling-related messages were the most frequent across departments, whereas symptoms and health concerns were second or third most common topics. In medical oncology and surgical oncology, topics on prescriptions and medications were more common compared with radiation oncology and gynecologic oncology. Despite concurrent institutional changes in self-scheduling systems, scheduling-related messages did not decrease over time. CONCLUSION: The substantial increase in patient portal messages, particularly scheduling-related inquiries, underscores the need for streamlined communication to reduce the burden on health care providers. These findings highlight the need for strategies to manage message volume and mitigate physician burnout, laying groundwork for artificial intelligence–driven future triage systems to improve message management and patient care.

Clinical reasoning is a fundamental skill in medical education that requires intensive faculty resources and deliberate practice. Here, we present the design and implementation of a novel adaptive Socratic tutor powered by large language models (LLMs) that facilitates case-based learning for medical trainees. Our system takes input from any structure of a published clinical case to create interactive and adaptive clinical scenarios where learners engage in realistic patient encounters with real-time feedback on their reasoning process. As a proof of concept, we demonstrate its use with the NEJM clinical pathological case series. This paper describes the architecture, knowledge representation, and educational design principles incorporated into our system, which we are releasing as an open-source tool for medical education. Our work demonstrates how LLMs can promote high-quality precision education in clinical reasoning and provide a structured assessment of the strengths and weaknesses of the learner.

Background: Early-stage breast cancer has the complex challenge of carrying a favorable prognosis with multiple treatment options, including breast-conserving surgery (BCS) or mastectomy. Social media is increasingly used as a source of information and as a decision tool for patients, and awareness of these conversations is important for patient counseling. Objective: The goal of this study was to compare sentiments and associated emotions in social media discussions surrounding BCS and mastectomy using natural language processing (NLP). Methods: Reddit posts and comments from the Reddit subreddit r/breastcancer and associated metadata were collected using pushshift.io. Overall, 105,231 paragraphs across 59,416 posts and comments from 2011 to 2021 were collected and analyzed. Paragraphs were processed through the Apache Clinical Text Analysis Knowledge Extraction System and identified as discussing BCS or mastectomy based on physician-defined Systematized Nomenclature of Medicine Clinical Terms (SNOMED CT) concepts. Paragraphs were analyzed with a VADER (Valence Aware Dictionary for Sentiment Reasoning) compound sentiment score (ranging from −1 to 1, corresponding to negativity or positivity) and GoEmotions scores (0‐1) corresponding to the intensity of 27 different emotions and neutrality. Results: Of the 105,231 paragraphs, there were 7306 (6.94% of those analyzed) paragraphs mentioning BCS and mastectomy (2729 and 5476, respectively). Discussion of both increased over time, with BCS outpacing mastectomy. The median sentiment score for all discussions analyzed in aggregate became more positive over time. In specific analyses by topic, positive sentiments for discussions with mastectomy mentions increased over time; however, discussions with BCS-specific mentions did not show a similar trend and remained overall neutral. Compared to BCS, conversations about mastectomy tended to have more positive sentiments. The most commonly identified emotions included neutrality, gratitude, caring, approval, and optimism. Anger, annoyance, disappointment, disgust, and joy increased for BCS over time. Conclusions: Patients are increasingly participating in breast cancer therapy discussions with a web-based community. While discussions surrounding mastectomy became increasingly positive, BCS discussions did not show the same trend. This mirrors national clinical trends in the United States, with the increasing use of mastectomy over BCS in early-stage breast cancer. Recognizing sentiments and emotions surrounding the decision-making process can facilitate patient-centric and emotionally sensitive treatment recommendations.

PURPOSE: We examined the effectiveness of proprietary and open large language models (LLMs) in detecting disease presence, location, and treatment response in pancreatic cancer from radiology reports. METHODS: We analyzed 203 deidentified radiology reports, manually annotated for disease status, location, and indeterminate nodules needing follow-up. Using generative pre-trained transformer (GPT)-4, GPT-3.5-turbo, and open models such as Gemma-7B and Llama3-8B, we employed strategies such as ablation and prompt engineering to boost accuracy. Discrepancies between human and model interpretations were reviewed by a secondary oncologist. RESULTS: Among 164 patients with pancreatic tumor, GPT-4 showed the highest accuracy in inferring disease status, achieving a 75.5% correctness (F1-micro). Open models Mistral-7B and Llama3-8B performed comparably, with accuracies of 68.6% and 61.4%, respectively. Mistral-7B excelled in deriving correct inferences from objective findings directly. Most tested models demonstrated proficiency in identifying disease containing anatomic locations from a list of choices, with GPT-4 and Llama3-8B showing near-parity in precision and recall for disease site identification. However, open models struggled with differentiating benign from malignant postsurgical changes, affecting their precision in identifying findings indeterminate for cancer. A secondary review occasionally favored GPT-3.5's interpretations, indicating the variability in human judgment. CONCLUSION: LLMs, especially GPT-4, are proficient in deriving oncologic insights from radiology reports. Their performance is enhanced by effective summarization strategies, demonstrating their potential in clinical support and health care analytics. This study also underscores the possibility of zero-shot open model utility in environments where proprietary models are restricted. Finally, by providing a set of annotated radiology reports, this paper presents a valuable data set for further LLM research in oncology.

Purpose We examined the effectiveness of proprietary and open Large Language Models (LLMs) in detecting disease presence, location, and treatment response in pancreatic cancer from radiology reports. Methods We analyzed 203 deidentified radiology reports, manually annotated for disease status, location, and indeterminate nodules needing follow-up. Utilizing GPT-4, GPT-3.5-turbo, and open models like Gemma-7B and Llama3-8B, we employed strategies such as ablation and prompt engineering to boost accuracy. Discrepancies between human and model interpretations were reviewed by a secondary oncologist. Results Among 164 pancreatic adenocarcinoma patients, GPT-4 showed the highest accuracy in inferring disease status, achieving a 75.5% correctness (F1-micro). Open models Mistral-7B and Llama3-8B performed comparably, with accuracies of 68.6% and 61.4%, respectively. Mistral-7B excelled in deriving correct inferences from “Objective Findings” directly. Most tested models demonstrated proficiency in identifying disease containing anatomical locations from a list of choices, with GPT-4 and Llama3-8B showing near parity in precision and recall for disease site identification. However, open models struggled with differentiating benign from malignant post-surgical changes, impacting their precision in identifying findings indeterminate for cancer. A secondary review occasionally favored GPT-3.5’s interpretations, indicating the variability in human judgment. Conclusion LLMs, especially GPT-4, are proficient in deriving oncological insights from radiology reports. Their performance is enhanced by effective summarization strategies, demonstrating their potential in clinical support and healthcare analytics. This study also underscores the possibility of zero-shot open model utility in environments where proprietary models are restricted. Finally, by providing a set of annotated radiology reports, this paper presents a valuable dataset for further LLM research in oncology.

Both medical care and observational studies in oncology require a thorough understanding of a patient’s disease progression and treatment history, often elaborately documented within clinical notes. As large language models (LLMs) are being considered for use within medical workflows, it becomes important to evaluate their potential in oncology. However, no current information representation schema fully encapsulates the diversity of oncology information within clinical notes, and no comprehensively annotated oncology notes exist publicly, thereby limiting a thorough evaluation. We curated a new fine-grained, expert-labeled dataset of 40 deidentified breast and pancreatic cancer progress notes at the University of California, San Francisco, and assessed the abilities of three recent LLMs (GPT-4, GPT-3.5-turbo, and FLAN-UL2) in zero-shot extraction of detailed oncological information from two narrative sections of clinical progress notes. Model performance was quantified with BLEU-4, ROUGE-1, and exact-match (EM) F1 score metrics. Our team annotated 9028 entities, 9986 modifiers, and 5312 relationships. The GPT-4 model exhibited overall best performance, with an average BLEU score of 0.73, an average ROUGE score of 0.72, an average EM F1 score of 0.51, and an average accuracy of 68% (expert manual evaluation on subset). Notably, GPT-4 was proficient in tumor characteristic and medication extraction and demonstrated superior performance in advanced reasoning tasks of inferring symptoms due to cancer and considerations of future medications. Common errors included partial responses with missing information and hallucinations with note-specific information. By developing a comprehensive schema and benchmark of oncology-specific information in oncology notes, we uncovered both the strengths and the limitations of LLMs. Our evaluation showed variable zero-shot extraction capability among the GPT-3.5-turbo, GPT-4, and FLAN-UL2 models and highlighted a need for further improvements, particularly in complex medical reasoning, before performing reliable information extraction for clinical research and complex population management and documenting quality patient care. (Funded by the National Institute of Health, Food and Drug Administration and others.)

Electronic health records (EHRs) provide meaningful knowledge of drug‐related adverse events (AEs) that are not captured in standard drug development and postmarketing surveillance. Using variables obtained from EHR data in the University of California San Francisco de‐identified Clinical Data Warehouse, we aimed to evaluate the potential of machine learning to predict two hematological AEs, thrombocytopenia and anemia, in a cohort of patients treated with linezolid for 3 or more days. Features for model input were extracted at linezolid initiation (index), and outcomes were characterized from index to 14 days post‐treatment. Random forest classification (RFC) was used for AE prediction, and reduced feature models were evaluated using cumulative importance (cImp) for feature selection. Grade 3+ thrombocytopenia and anemia occurred in 31% of 2,171 and 56% of 2,170 evaluable patients, respectively. Of the total 53 features, as few as 7 contributed at least 50% cImp, resulting in prediction accuracies of 70% or higher and area under the receiver operating characteristic curves of 0.886 for grade 3+ thrombocytopenia and 0.759 for grade 3+ anemia. Sensitivity analyses in strictly defined patient subgroups revealed similarly high predictive performance in full and reduced feature models. A logistic regression model with the same 50% cImp features showed similar predictive performance as RFC and good concordance with RFC probability predictions after isotonic calibration, adding interpretability. Collectively, this work demonstrates potential for machine learning prediction of AE risk in real‐world patients using few variables regularly available in EHRs, which may aid in clinical decision making and/or monitoring.