Tellen D. Bennett

Objectives: Comorbidity algorithms derived from International Classification of Diseases (ICD) codes are central to risk adjustment and cohort characterization, but existing software inconsistently handles mixed ICD-9/10 data and typically operates at the encounter level. We developed medicalcoder to address these limitations. Materials and Methods: medicalcoder is an R package with an internal ICD database and unified interface for applying variants of the Charlson, Elixhauser, and Pediatric Complex Chronic Conditions (PCCC) algorithms. It supports full and compact ICD codes, mixed ICD versions, present-on-admission (POA) and primary diagnosis indicators, and encounter-level and cumulative longitudinal flagging. Results: The medicalcoder package requires only R ≥ 3.5.0; no additional packages, internet access, or external data are required. Outputs are consistent with published reference implementations. Longitudinal flagging identifies additional comorbidity flags under a carry-forward assumption while accounting for POA or technology-dependent logic. Conclusions: medicalcoder provides a unified, reproducible framework for ICD-based comorbidity assessment in modern clinical datasets.

OBJECTIVES: Sepsis is a heterogeneous syndrome resulting from a complex interaction between the infectious agent and the host response. In adults with sepsis, longitudinal temperature measurements have been used to identify phenotypes with distinct clinical characteristics, immune profiles, and outcomes. Our objective was to develop and validate temperature trajectory-based phenotypes in critically ill children with sepsis. We hypothesized that the temperature trajectory-based phenotypes would be independently associated with clinical outcomes. DESIGN: Retrospective observational cohort study. SETTING: A total of 13 PICUs in the United States from January 1, 2012, to January 1, 2018. PATIENTS: Children (age 0–18 y) with community-acquired sepsis meeting Phoenix sepsis criteria admitted to a PICU directly or via an emergency department, who survived to 24 hours and had greater than or equal to 4 temperature measurements in the first 24 hours of admission were included. INTERVENTIONS: None. MEASUREMENTS AND MAIN RESULTS: A total of 11,566 patients met all criteria for study inclusion. A total of 2327 patients (20%) developed the primary outcome of persistent multiple organ dysfunction syndrome (MODS) assessed at day 7 of admission. We used group-based trajectory modeling to identify temperature trajectory-based phenotypes. Three groups were identified: “hyperthermic” (35%), “normothermic” (59%), and “hypothermic” (6%), which were reproducible and had similar outcomes on a validation set. The “hypothermic” trajectory-based phenotype group had the highest odds of persistent MODS odds ratio (OR), 5 (95% CI, 4–6), and ICU mortality OR, 11 (95% CI, 9–15). In an exploratory analysis, hypothermia (< 35°C) at the time of PICU admission was also associated with persistent MODS and ICU mortality. The association between “hypothermic” trajectory-based phenotype and poor outcome remained after adjusting for hypothermia at admission, severity of illness, age, and technology dependence. CONCLUSIONS: We developed and validated temperature trajectory-based phenotypes of pediatric sepsis in a large multicenter cohort. The “hypothermic” trajectory-based phenotype was independently associated with worse clinical outcomes.

Clinical decisions for determining optimal patient-specific interventions are complicated prediction tasks that rely on health care professionals’ understanding of physiological mechanisms and their dynamics. These decisions are challenged by (a) observational data sparsity and (b) patient heterogeneity. Here, we focus on estimating and forecasting specific physiological properties—that are not explicitly present in clinical observations—to provide additional features using only data available bedside at the time of decision-making. Mechanistic models of physiological system(s), e.g., physiological ordinary differential equation (ODE) models, provide pathways to compensate for data sparsity by synchronizing the model with observations of an individual patient using data assimilation (DA). However, DA used in a standard computational workflow to estimate constant model parameters from presently-known data is less effective at optimizing state forecasts of the model governed by physiological processes that evolve before new observations are available. Stated simply, we cannot forecast the future evolution of the model because we cannot forecast model parameters. To support next-generation clinical decision support, we develop a new DA and machine learning (ML) hybrid pipeline to estimate and forecast individual future physiological processes by forecasting ODE model parameters. This pipeline overcomes model and DA workflow limitations by stacking a DA-estimated posterior empirical distribution of physiological parameters with longitudinal ML forecasting models. We work within the context of glycemic management in an ICU using EHR data to construct and test a use case. We use synthetic data and real-world clinical data to validate the integrated pipeline and quantify uncertainties.

BACKGROUND Clinical decision support (CDS) tools within electronic health records have the potential to improve evidence-based care but often fall short. One potential approach to improve CDS effectiveness is to extend CDS tools to include clinician-specific personalization by leveraging data on clinician behavior in the EHR. OBJECTIVE The objective of this study is to determine clinician attitudes towards CDS tools in general, review common concerns, and anticipate in what ways a clinician-personalized CDS approach can solve issues clinicians have with CDS tools. METHODS Using the case example of guideline directed medical treatment for heart failure, we conducted a qualitative study using semi-structured interviews with outpatient clinicians in primary care and cardiology settings. Participants were purposively sampled to capture diverse perspectives across specialties, practice settings, and clinician types. Interviews were analyzed using a pragmatic thematic approach. RESULTS Nineteen clinicians participated. Across specialties, clinicians viewed CDS as a valuable cognitive support tool, particularly for managing complex guidelines and reducing mental workload. However, traditional CDS were consistently described as burdensome due to high alert volume, generic content, poor timing, and workflow disruption, contributing to alert fatigue and reduced trust. Clinician-personalized CDS was endorsed as a potential solution, with participants highlighting its ability to tailor information based on clinician knowledge and practice patterns, reduce redundant content, and enhance relevance. However, clinicians also raised concerns regarding the ability of algorithms to capture clinical nuance, the validity and transparency of clinician-level data, and the risk of perceived loss of autonomy or punitive feedback. CONCLUSIONS Clinician-personalized CDS represents a promising extension of current CDS approaches by incorporating clinician context to improve relevance and usability. While early perceptions are favorable, important challenges related to trust, feasibility, and implementation complexity remain. Further evaluation is needed to determine whether clinician-personalization improves clinical outcomes and clinician experience, and to identify when and how this approach provides meaningful value within health systems.

BACKGROUND: Acute kidney injury (AKI) is common in critically ill children and is associated with high morbidity and mortality. Risk prediction models designed for clinical decision support implementation can facilitate early identification and proactive mitigation of AKI risk. Existing models have primarily been validated using single-center data, partly because of the lack of appropriately detailed multicenter datasets. OBJECTIVE: To determine the performance of a single-center model to predict new AKI at 72 hours of ICU admission in children across two multicenter datasets and refine this model to improve prediction performance while maintaining acceptable alert burden. DERIVATION AND VALIDATION COHORTS: We analyzed two datasets: the Pediatric Learning Health System Network–Virtual Pediatric Systems (PEDSNET-VPS) dataset, created through the linkage of PEDSnet electronic health record (EHR) extraction with VPS (LLC, http://www.myvps.org), and the PICU Data Collaborative dataset, created through EHR extraction and harmonization from eight participating institutions. We divided each dataset into a derivation and test split. PREDICTION MODEL: We first recalibrated an existing single-center model and measured discrimination (area under the receiver operating characteristic curve [AUROC] and area under the precision-recall curve [AUPRC]) and performance at multiple cutpoints. We next added features available at 12 hours of ICU admission, optimizing by precision and recall. We measured discrimination and performance at multiple cutpoints and identified the features contributing most to the risk score. RESULTS: In total we analyzed 186,540 ICU admissions. We found early AKI by serum creatinine criteria within 72 hours of admission in 2.2–2.7%. Initial recalibration of an existing single-center model demonstrated poor discrimination (AUROC 0.65–0.78; AUPRC 0.10–0.12). Following the addition of new features, the model had higher AUROC (0.80–0.88) and AUPRC (0.13–0.22). CONCLUSIONS: In this first use of two new multicenter datasets, we found improved performance in a model designed using features available at 12 hours of ICU admission, balancing sensitivity and precision to predict patients at risk for AKI development.

Background: Large language models (LLMs) are increasingly used in medication-related tasks despite limited evidence supporting their accuracy and safety. LLMs are vulnerable to adversarial attacks and may confabulate in response to fabricated information in the prompt. Given the substantial risk of harm from medication errors, the purpose of this study was to establish a benchmark task to evaluate LLM performance in distinguishing fabricated from real medications. Methods: Two datasets (brand and generic), each consisting of 250 medication lists were developed, each including 4-6 medications and one fabricated medication (specifically a Pokémon character), with complete dosing information provided. Six LLMs were evaluated (GPT-5-Chat, GPT-4o-mini, DrugGPT, Gemma-3-27B-IT, Llama-3.3-70B-Instruct, and Qwen3-32B). For each medication list, models were queried for dosing information and disease indication across three experimental conditions: standard decoding, standard decoding with mitigation prompting, and deterministic decoding (temperature = 0). Each experiment was performed in triplicate. Outputs were deemed confabulations if the LLM failed to recognize and reject the fabricated medication. The primary outcome was the rate of confabulations. Exact paired-permutation tests were used to compare confabulation rates across LLMs and prompting approaches. Results: Confabulation rates under standard conditions ranged from 2.7% to 99.6%. DrugGPT demonstrated the lowest baseline confabulation rates (2.7-6.4%), whereas other models demonstrated substantially higher rates (47.2-99.6%). Mitigation prompting significantly reduced confabulation rates compared to standard and deterministic decoding across all tasks and datasets (p < 0.001): GPT-5-Chat achieved confabulation rates of 0% with mitigation prompting from baseline rates of 66.4-88.5%. Deterministic decoding did not substantially impact confabulation rates. Conclusions: LLMs performing medication-related tasks are susceptible to confabulation in response to invalid or fabricated inputs, raising important safety concerns for clinical use. While mitigation prompting reduced confabulation rates, this study underscores the importance of robust safeguards when deploying LLMs in medication-related decision-making.

Background: The purpose of this study was to develop a series of benchmarks specifically for large language model (LLM) performance in the identification of drug-drug interactions (DDIs). Methods: We evaluated six LLMs using a clinician-designed dataset of 750 DDI scenarios. Tasks included: (1) a pointwise two-drug classification task (2) a pairwise three-drug discrimination task and (3) a listwise 4–6 drug selection task. Zero-shot prompting was applied and performance was assessed using precision, recall, F1 score, accuracy, and self-consistency. Results: Model performance varied by task structure and interaction severity. GPT-5-Chat demonstrated the highest recall and F1 score in the pointwise task (80.7% and 80.2%), whereas DrugGPT achieved superior accuracy in the pairwise (93.7%) and listwise task (92.6%). All models had improved performance with more severe compared to less severe DDIs (accuracy 72.0-99.2% vs. 58.3-86.7%). All models tended to have higher self-consistency in the pairwise and listwise experiments compared to the pointwise experiment (68.8-93.6% vs. 31.0-74.2%). No model exhibited uniformly high reliability across all tasks. Conclusions: Current LLMs show inconsistent capabilities in identifying DDIs. Performance degrades as reasoning space expands with limited stability. These findings emphasize the need for multi-format evaluation frameworks and reliability-aware assessment when considering LLMs for medication-safety applications.

Background: For large language models (LLMs) to reach their potential as information technology tools that make medication use safer, clinically relevant benchmarks capable of automated grading and designed specifically to measure the performance of LLMs for medication tasks are required. The purpose of this study was to design a suite of benchmarking tests reflective of Comprehensive Medication Management (CMM; the standard of care for medication optimization) and quantify the baseline performance of the latest LLMs. Methods: We established six benchmarks representing critical stages of the CMM process: drug formulation matching, drug order (sig) generation, drug route matching, drug-drug interaction identification, renal dose identification, and drug-indication matching. For each benchmark, we curated a clinician-annotated dataset comprising 250 standardized input-output pairs including both inpatient and outpatient medications. We evaluated the clinical knowledge retrieval capabilities of three LLMs: GPT-4o-mini, MedGemma-27B, and LLaMA3-70B. We employed a zero-shot prompting strategy, excluding in-context examples, to assess the models’ internal clinical knowledge rather than their few-shot learning potential. To check reliability, each model was run three times using a temperature of 0.7 (a mid-range value of an LLM setting controlling text generation randomness). Performance was assessed using task-specific evaluation metrics including precision (positive predictive value), recall (sensitivity), F1-score, accuracy, and correctness consistency across trials. Results: Across six benchmarks, LLaMA3-70B demonstrated the highest performance in four tasks: drug-formulation matching (F1, 54.0% [95 CI: 50.1-58]), drug-order generation (accuracy, 88.0%), drug-route identification (F1, 74.3% [95 CI: 71-78]), and drug-indication identification (accuracy, 97.6% [95 CI: 95.6-99.2]). In the drug–drug interaction task, GPT-4o-mini achieved the highest overall accuracy (70.4% [95 CI: 64.8-75.7]). For renal dose–adjustment identification, GPT-4o-mini demonstrated the highest F1 score (83.3% [95 CI: 77.6-88]). Correctness-consistency scores ranged from 8.0% to 97.6% across benchmarks, with no model exhibiting uniformly superior consistency. Conclusions: Model performance varied substantially across medication-related tasks. LLaMA3-70B demonstrated promising baseline performance in tasks involving formulation, ordering, route, and indication. GPT-4o-mini showed potential advantages in drug–drug interaction detection and renal dose adjustment. These findings underscore the need for task-specific evaluation when deploying models for medication-focused clinical decision support.

OBJECTIVE: To determine the accuracy of a custom version of the generative pretrained transformer (GPT)-4o large language model (LLM) in identifying PICU admissions with vs. without bacterial pneumonia using clinical notes. DESIGN: In this retrospective cohort study, the GPT-4o model was provided guidance on our institution’s pneumonia diagnosis practices through a custom prompt and instructed to analyze PICU provider notes from the first 2 calendar days of PICU admission to identify bacterial pneumonia diagnoses. Diagnoses from the manually curated Virtual Pediatric Systems (VPS) Registry were used as the gold standard. SETTING: A 48-bed, academic, quaternary care PICU. PATIENTS: Children 3 months old to 18 years old admitted to the PICU from January 1, 2023, to December 31, 2023. INTERVENTIONS: None. MEASUREMENTS AND MAIN RESULTS: GPT-4o analyzed 10,081 notes from 3,317 PICU admissions over 5.0 minutes (mean 0.03 s per note). Of the 3317 study encounters, 481(14.5%) had a VPS admission pneumonia diagnosis. GPT-4o accurately classified 3143 of 3317 (94.8%) encounters. In a post hoc adjudication analysis, a blinded PICU attending reviewed patient charts with VPS-GPT discordant classifications. The GPT-4o classification matched that of the blinded PICU attending in 125 of 174 (71.8%) of such encounters. The most common reason for incorrect classification by GPT-4o was that a pneumonia diagnosis was listed in the initial notes but later rescinded when a different diagnosis was identified. CONCLUSIONS: The GPT-4o LLM was able to accurately and rapidly identify critically ill children with vs. without bacterial pneumonia. This study suggests similar tools could be developed to automate and accelerate processes typically requiring manual chart review.

Objectives: We discuss challenges using computational modeling approaches for personalized prediction in clinical practice to predict treatment response for rare diseases treated by novel therapies using clinical oncology as an example context. Several challenges are discussed, including data scarcity, data sparsity, and difficulties in establishing interdisciplinary teams. Machine learning (ML), mechanistic modeling (MM), and hybrid modeling (HM) are discussed in the context of these challenges. Materials and Methods: We present an HM approach, combining ML and MM techniques for improved personalized model estimation in the context of chimeric antigen receptor T-cell therapy for aggressive lymphoma. Results: The HM approach improved the root mean squared error by 61.27±23.21% compared to using MM alone (MM: 2.36*105∓1.68*105and HM: 9.57*104∓8.37*104, where the units are in cells), computed from 13 patients included in this study. Discussion: By exploiting the complementary strengths of ML and MM approaches, the developed HM method addresses common limitations such as data scarcity and sparsity in medical settings, especially common for rare diseases. Conclusion: The HM techniques are likely required to overcome data scarcity and sparsity issues in broad medical settings. Developing these techniques requires dedicated interdisciplinary teams.