Joseph M. Plasek

Background: Electronic frailty indices (eFIs) derived from structured electronic health record (EHR) data may incompletely capture functional deficits. Large language models (LLMs) can extract frailty-related information from clinical notes, but whether improved frailty ascertainment improves prognostic performance is uncertain. Objective: To compare frailty ascertainment and prognostic performance between standard and LLM-enhanced eFIs among hospitalized adults. Design: Retrospective cohort study Participants: Adults aged ≥50 years enrolled in two studies at an academic medical center and affiliated community hospital between February 2022 and April 2024. Main Measures: A validated 35-deficit diagnosis-based eFI and an LLM-enhanced eFI incorporating deficits extracted from physical therapy (PT) documentation using GPT-4o. Frailty was defined using an index threshold of ≥0.20. The primary outcome was a composite of discharge to a facility, readmission, or death within 90 days. Multivariable logistic regression models assessed associations with the outcome, and discrimination was compared using area under the receiver operating characteristic curve (AUROC). Secondary analyses examined threshold-based classification and PT-documentation availability; prespecified sensitivity analyses evaluated population and outcome restrictions. Results: Among 616 hospitalizations, mean (SD) age was 70.9 (10.2) years, 329 (53.4%) had PT documentation, and 320 (51.9%) experienced the composite outcome. Standard and LLM-enhanced eFIs classified 216 (35.1%) and 362 (58.8%) hospitalizations as frail, respectively. Both were independently associated with the outcome (standard eFI: adjusted OR, 1.98 [95% CI, 1.39–2.83]; LLM-enhanced eFI: adjusted OR, 2.46 [95% CI, 1.75–3.47]). Discrimination did not differ significantly (AUROC, 0.636 vs 0.660; difference, 0.024 [95% CI, −0.006 to 0.053]; p =0.12). LLM enhancement increased sensitivity (42.5% to 69.4%) but decreased specificity (73.0% to 52.7%). Conclusions: LLM enhancement substantially increased frailty ascertainment and sensitivity for adverse outcomes but did not significantly improve prognostic discrimination. Thus, LLM-enhanced frailty assessment may depend on whether the intended use prioritizes broad identification of functional vulnerability or prediction of adverse post-hospital outcomes.

Background: Systematic collection of social determinants of health (SDoH) data remains inconsistent across health care settings, despite its critical impact on patient outcomes. Large language model–powered chatbots offer promise for scalable SDoH data collection, but rigorous, feasible evaluation methods for patient-facing applications are lacking. Objective: This study aimed to describe an efficient, iterative, multidisciplinary approach for developing and evaluating a patient-facing SDoH chatbot using synthetic data and case simulation, with the goal of optimizing both chatbot performance and the evaluation rubric prior to clinical deployment. Methods: A 10-criterion evaluation rubric was adapted from established health care AI frameworks and applied to 27 synthetic clinical scenarios representing diverse SDoH profiles. Scenarios were role-played by a licensed clinical social worker, and chatbot-patient interactions were rated by 3 members of the research team that were multidisciplinary experts: a social worker, a nurse practitioner, and a physician. Quantitative analysis used percent agreement and Fleiss κ to characterize chatbot performance and rater consensus, with percent agreement selected due to the high prevalence of ceiling effects in several domains. Qualitative analysis synthesized rater feedback to guide iterative refinement of both chatbot prompts and rubric domains. Results: Across 27 simulated cases, the chatbot received high proportions of positive ratings for accurate interpretation (agreement=0.98%, 95% CI 0.91‐0.99), communication quality, and cultural sensitivity (agreement=0.99%, 95% CI 0.93‐1.00), and appropriately adaptive questioning (agreement=0.99%, 95% CI 0.93‐1.00). Lower performance was observed in domain focus and completeness (agreement=0.51%, 95% CI 0.40‐0.61), completeness of data capture (agreement=0.59%, 95% CI 0.48‐0.69; Fleiss κ=0.18), and safety (agreement=0.69%, 95% CI 0.58‐0.78; Fleiss κ=−0.04), prompting targeted adaptations. Qualitative feedback highlighted the importance of distinguishing screening from clinical interviewing capabilities and informed the refinement of the rubric, including clarifying the definition of safety to focus on recognition of physical and mental health emergencies. Conclusions: This study describes a formative feasibility approach for iterative refinement of a patient-facing SDoH chatbot and its evaluation rubric using synthetic case simulation. Future work will include independent external raters, patient stakeholders, repeated scenario testing, and prospective clinical evaluation.

BACKGROUND Systematic collection of social determinants of health (SDoH) data remains inconsistent across healthcare settings, despite its critical impact on patient outcomes. Large language model (LLM)-powered chatbots offer promise for scalable SDoH data collection, but rigorous, feasible evaluation methods for patient-facing applications are lacking. OBJECTIVE To describe an efficient, iterative, multidisciplinary approach for developing and evaluating a patient-facing SDoH chatbot using synthetic data and case simulation, with the goal of optimizing both chatbot performance and the evaluation rubric prior to clinical deployment. METHODS A 10-criterion evaluation rubric was adapted from established healthcare artificial intelligence (AI) frameworks and applied to 27 synthetic clinical scenarios representing diverse SDoH profiles. Scenarios were role-played by a licensed clinical social worker, and chatbot-patient interactions were independently rated by three multidisciplinary experts (social worker, nurse practitioner, physician). Quantitative analysis used descriptive statistics and percent agreement to characterize chatbot performance and rater consensus, with percent agreement selected due to high prevalence of ceiling effects in several domains. Qualitative analysis synthesized rater feedback to guide iterative refinement of both chatbot prompts and rubric domains. RESULTS The chatbot demonstrated robust performance in domains such as accurate interpretation (mean = 0.98, SD = 0.09), communication quality and cultural sensitivity (mean = 0.99, SD = 0.06), and adaptive questioning (mean = 0.99, SD = 0.06), with near-perfect rater agreement. Lower scores and greater variability were observed in completeness of data collection, systematic domain exploration, and safety, prompting targeted adaptations. Qualitative feedback highlighted the importance of distinguishing screening from clinical interviewing capabilities and informed the refinement of the rubric, including clarifying the definition of safety to focus on recognition of physical and mental health emergencies. CONCLUSIONS This study provides a practical, replicable blueprint for pre-deployment evaluation of patient-facing SDoH chatbots, balancing rigor with feasibility. The iterative, multidisciplinary approach enabled rapid identification and remediation of performance gaps, supporting responsible integration of AI into SDoH data collection. Explicit performance thresholds and rubric refinement are essential for protecting patient trust and safety, particularly in vulnerable populations. Future work will validate findings with real patient interactions and expand stakeholder involvement. CLINICALTRIAL N/A

Multiple imputation (MI) models can be improved with auxiliary covariates (ACs), but their performance in high-dimensional data remains unclear. We aimed to develop and compare high-dimensional MI (HDMI) methods using structured and natural language processing (NLP)–derived AC in studies with partially observed confounders. We conducted a plasmode simulation with acute kidney injury as outcome and simulated 100 cohorts with a null treatment effect, incorporating creatinine labs, atrial fibrillation (AFib), and other investigator-derived confounders in the outcome generation. Missingness was imposed on creatinine based on creatinine itself and AFib. Different HDMI candidate ACs were created using structured and NLP-derived features, and we mimicked scenarios where AFib was unobserved by omitting it from all analyses. Using the least absolute shrinkage and selection operator, we selected HDMI covariates for MI and propensity score models. The treatment effect was estimated after propensity score matching in MI datasets, and HDMI methods were compared to baseline imputation and complete case analysis. High-dimensional MI using claims data showed the lowest bias (0.072). Combining claims and sentence embeddings led to an improvement in the efficiency with a root mean square error (RMSE) of 0.173 and 94% coverage. Natural language processing–derived AC alone did not outperform baseline MI. High-dimensional MI approaches may decrease bias in studies where confounder missingness depends on unobserved factors.

Background Adverse drug events (ADEs) are understudied in the ambulatory care setting. We aim to estimate the prevalence and characteristics of ADEs in outpatient care using electronic health records (EHRs). Methods This cross-sectional study included EHR data for patients who had an outpatient encounter at an academic medical center from 1 October 2018 through 31 December 2019. We developed a stratified sampling strategy based on a comprehensive set of 994 ADE-related International Classification of Disease (ICD-10) codes to identify clinical encounters and notes likely to contain ADEs. Within each ICD-10 likelihood group, clinical notes were randomly sampled and annotated for present or possible ADE-drug relationships and severity. The overall estimated population prevalence of ADEs presenting in the outpatient setting was calculated. The generalizability of the findings was assessed by comparing ICD-10 code frequencies against a large commercial database. Results The study included 3126 notes (unique patient encounters) from 2882 unique patients. Of these, 1383 patient encounters (44.2%) had a present or possible ADE documented (6308 mentions). Of the 6038 ADEs mentioned, 14.1% were hypersensitivity reactions, 1.1% were life-threatening, 22.4% were serious, and 60.4% were significant. Main causal agents included anti-infectives (19.3%), central nervous system agents (12.8%), and cardiovascular agents (11.5%). The overall prevalence of present ADEs mentioned in the clinical notes was estimated to be 1.97 per 100 patient encounters (or 2.52 per 100 patient encounters when possible ADEs are included). Conclusions This study identified the overall population prevalence per encounter of ADEs in the outpatient population by leveraging ICD-10 codes and investigating ADEs documented in clinical notes. Understanding the ADE characteristics in a large corpus of outpatient documentation advances pharmacovigilance knowledge, enhancing the detection, monitoring, and prevention of ADEs in ambulatory care.

Objectives: Generative large language models (LLMs) are a subset of transformers-based neural network architecture models. LLMs have successfully leveraged a combination of an increased number of parameters, improvements in computational efficiency, and large pre-training datasets to perform a wide spectrum of natural language processing (NLP) tasks. Using a few examples (few-shot) or no examples (zero-shot) for prompt-tuning has enabled LLMs to achieve state-of-the-art performance in a broad range of NLP applications. This article by the American Medical Informatics Association (AMIA) NLP Working Group characterizes the opportunities, challenges, and best practices for our community to leverage and advance the integration of LLMs in downstream NLP applications effectively. This can be accomplished through a variety of approaches, including augmented prompting, instruction prompt tuning, and reinforcement learning from human feedback (RLHF). Target Audience: Our focus is on making LLMs accessible to the broader biomedical informatics community, including clinicians and researchers who may be unfamiliar with NLP. Additionally, NLP practitioners may gain insight from the described best practices. Scope: We focus on 3 broad categories of NLP tasks, namely natural language understanding, natural language inferencing, and natural language generation. We review the emerging trends in prompt tuning, instruction fine-tuning, and evaluation metrics used for LLMs while drawing attention to several issues that impact biomedical NLP applications, including falsehoods in generated text (confabulation/hallucinations), toxicity, and dataset contamination leading to overfitting. We also review potential approaches to address some of these current challenges in LLMs, such as chain of thought prompting, and the phenomena of emergent capabilities observed in LLMs that can be leveraged to address complex NLP challenge in biomedical applications.

Identifying consumer health informatics (CHI) literature is challenging. To recommend strategies to improve discoverability, we aimed to characterize controlled vocabulary and author terminology applied to a subset of CHI literature on wearable technologies. To retrieve articles from PubMed that addressed patient/consumer engagement with wearables, we developed a search strategy of textwords and Medical Subject Headings (MeSH). To refine our methodology, we used a random sample of 200 articles from 2016 to 2018. A descriptive analysis of articles (N = 2522) from 2019 identified 308 (12.2%) CHI-related articles, for which we characterized their assigned terminology. We visualized the 100 most frequent terms assigned to the articles from MeSH, author keywords, CINAHL, and Engineering Databases (Compendex and Inspec together). We assessed the overlap of CHI terms among sources and evaluated terms related to consumer engagement. The 308 articles were published in 181 journals, more in health journals (82%) than informatics (11%). Only 44% were indexed with the MeSH term “wearable electronic devices.” Author keywords were common (91%) but rarely represented consumer engagement with device data, eg, self-monitoring (n = 12, 0.7%) or self-management (n = 9, 0.5%). Only 10 articles (3%) had terminology from all sources (authors, PubMed, CINAHL, Compendex, and Inspec). Our main finding was that consumer engagement was not well represented in health and engineering database thesauri. Authors of CHI studies should indicate consumer/patient engagement and the specific technology investigated in titles, abstracts, and author keywords to facilitate discovery by readers and expand vocabularies and indexing.