Zidu Xu

LLMs are increasingly used to mediate patient communication, yet scalable evaluation of their safety, accuracy, and communication quality remains an open problem. LLM judges have emerged as automated evaluators, but whether they can holistically replicate human expert judgment is unvalidated. Informed consent for clinical trials presents a demanding case for such validation because it requires conveying complex information to lay audiences under ethical and safety constraints. We developed a stakeholder-informed seven-criterion evaluation rubric spanning safety, reliability, and communication quality. Clinician reference ratings showed strong interrater reliability across all criteria. We validated the rubric on the Informed CONsent Benchmark (ICON-Bench) and benchmarked 19 LLM judges across multiple implementation strategies. LLM judges achieved strong clinician agreement for safety screening and factual verification (Spearman ρ > 0.80) but weaker agreement for communication quality ( ρ < 0.60). Safety-specialized guard models underperformed general-purpose models. Patient advocates rated communication quality lower than both clinicians and LLM judges. These findings support LLM judges for scalable patient communication oversight while demonstrating the need for recalibration to patient-centered evaluation standards.

Objective: Home healthcare (HHC) clinical notes contain critical infection indicators that clinicians need in structured “indicator + context” pairs. Data sparsity and limited computing resources hinder automated extraction in decentralized HHC settings. This study developed and evaluated a resource-efficient pipeline using instruction-tuned, moderate-sized large language models (LLMs) to address these barriers. To address the data sparsity challenge, we also assessed the impact of a targeted LLM-based data augmentation strategy. Materials and Methods: An expert-defined schema of 26 infection indicator categories was developed. We expanded the training set using a 3-stage workflow: targeted annotation, context mutation, and synthetic generation. We adapted 2 moderate-sized models (Gemma-12B and Qwen-14B) via Quantized Low-Rank Adaptation (QLoRA). We compared them to a larger-sized, prompted model and a smaller-sized, fully fine-tuned LLM. We evaluated all models on a held-out test set using partial micro-averaged F1 score, output reliability metrics, and qualitative error analysis. Results: Instruction-tuned moderate-sized LLMs outperformed both baselines. The top-performing model, augmented Gemma-12B, achieved a partial micro-averaged F1 score of 0.879. LLM-based data augmentation enhanced overall performance, improving the identification of rare indicators and the interpretation of negations. The best model maintained a partial F1 score above 0.750 across all indicator categories. It also showed high format adherence, confirming its ability to generate reliable structured outputs. Discussion: Instruction-tuning moderate-sized LLMs with QLoRA and targeted data augmentation enables high-accuracy extraction of infection indicators from HHC notes. Conclusion: This resource-efficient pipeline provides a scalable foundation for automated infection surveillance in healthcare settings with limited resources.

Background: Post-acute care (PAC) utilization following sepsis hospitalization remains understudied, particularly concerning racial and ethnic and urban-rural disparities. Objectives: To examine trends and disparities in PAC utilization after sepsis hospitalization, focusing on race, ethnicity, and rurality. Methods: A comprehensive search of databases (PubMed, CINAHL, Embase, Web of Science, and Scopus) was conducted for eligible studies using data through March 2020. The Social Ecological Model guided the review. Results: Eleven studies met inclusion criteria. Our synthesis found a discontinuous increase in PAC use, with a shift from home discharges toward greater use of nursing homes and home health care after 2006. White patients had higher PAC utilization than racial and ethnic minority individuals. Rural and urban non-teaching hospitals discharged more sepsis survivors to long-term care hospitals, while urban teaching hospitals had more discharges to HHC. Conclusion: This review establishes a pre-reform, pre-pandemic baseline for PAC utilization among sepsis survivors. Despite overall gains, disparities in PAC utilization persist by race, ethnicity, and hospital type. As payment and care delivery models have evolved since 2016, future research should leverage this historical baseline to assess the impact of new policies on equitable PAC access for sepsis survivors.

PURPOSE: Large language models (LLMs) have demonstrated remarkable versatility in oncology applications, such as cancer staging and survival analysis. Despite their potential, ethical concerns such as data privacy breaches, bias in training data, lack of transparency, and risks associated with erroneous outputs pose significant challenges to their adoption in high-stakes oncology settings. Therefore, we aim to explore the ethical challenges associated with LLM-based applications in oncology and evaluate emerging techniques designed to address these issues. METHODS: Following the Preferred Reporting Items for Systematic Reviews and Meta-Analyses framework, a systematic review was conducted to evaluate publications related to ethical issues of LLMs in oncology across eight academic databases (eg, PubMed, Web of Science, and Embase) between January 1, 2019, and December 31, 2024. RESULTS: The search retrieved 4,319 published articles, of which 65 publications were preserved and included in our analysis. We identified six prevalent ethical challenges in oncology, including trust, equity, privacy, transparency, nonmaleficence, and accountability. We then evaluated emerging technical solutions to mitigate ethical challenges and summarized evaluation metrics used to assess these solutions' effectiveness. CONCLUSION: This review provides actionable recommendations for responsibly deploying LLMs in oncology, ensuring adherence to ethical guidelines, and fostering improved patient outcomes. By bridging technical and clinical perspectives, this review offers a foundational framework for advancing ethical artificial intelligence applications in oncology and highlights areas for future research.

Objectives: This study aims to (1) review machine learning (ML)-based models for early infection diagnostic and prognosis prediction in post-acute care (PAC) settings, (2) identify key risk predictors influencing infection-related outcomes, and (3) examine the quality and limitations of these models. Materials and Methods: PubMed, Web of Science, Scopus, IEEE Xplore, CINAHL, and ACM digital library were searched in February 2024. Eligible studies leveraged PAC data to develop and evaluate ML models for infection-related risks. Data extraction followed the CHARMS checklist. Quality appraisal followed the PROBAST tool. Data synthesis was guided by the socio-ecological conceptual framework. Results: Thirteen studies were included, mainly focusing on respiratory infections and nursing homes. Most used regression models with structured electronic health record data. Since 2020, there has been a shift toward advanced ML algorithms and multimodal data, biosensors, and clinical notes being significant sources of unstructured data. Despite these advances, there is insufficient evidence to support performance improvements over traditional models. Individual-level risk predictors, like impaired cognition, declined function, and tachycardia, were commonly used, while contextual-level predictors were barely utilized, consequently limiting model fairness. Major sources of bias included lack of external validation, inadequate model calibration, and insufficient consideration of data complexity. Discussion and Conclusion: Despite the growth of advanced modeling approaches in infection-related models in PAC settings, evidence supporting their superiority remains limited. Future research should leverage a socio-ecological lens for predictor selection and model construction, exploring optimal data modalities and ML model usage in PAC, while ensuring rigorous methodologies and fairness considerations.

Background: The healthcare industry increasingly values high‐quality and personalized care. Patients with heart failure (HF) receiving home health care (HHC) often experience hospitalizations due to worsening symptoms and comorbidities. Therefore, close symptom monitoring and timely intervention based on risk prediction could help HHC clinicians prevent emergency department (ED) visits and hospitalizations. This study aims to (1) describe important variables associated with a higher risk of ED visits and hospitalizations in HF patients receiving HHC; (2) map data requirements of a clinical decision support (CDS) tool to the exchangeable data standard for integrating a CDS tool into the care of patients with HF; (3) outline a pipeline for developing a real‐time artificial intelligence (AI)‐based CDS tool.Methods: We used patient data from a large HHC organization in the Northeastern US to determine the factors that can predict ED visits and hospitalizations among patients with HF in HHC (9362 patients in 12,223 care episodes). We examined vital signs, HHC visit details (e.g., the purpose of the visit), and clinical note–derived variables. The study identified critical factors that can predict ED visits and hospitalizations and used these findings to suggest a practical CDS tool for nurses. The tool's proposed design includes a system that can analyze data quickly to offer timely advice to healthcare clinicians.Results: Our research showed that the length of time since a patient was admitted to HHC and how recently they have shown symptoms of HF were significant factors predicting an adverse event. Additionally, we found this information from the last few HHC visits before the occurrence of an ED visit or hospitalization were particularly important in the prediction. One hundred percent of clinical demographic profiles from the Outcome and Assessment Information Set variables were mapped to the exchangeable data standard, while natural language processing–driven variables couldn't be mapped due to their nature, as they are generated from unstructured data. The suggested CDS tool alerts nurses about newly emerging or rising risks, helping them make informed decisions.Conclusions: This study discusses the creation of a time‐series risk prediction model and its potential CDS applications within HHC, aiming to enhance patient outcomes, streamline resource utilization, and improve the quality of care for individuals with HF.Clinical Relevance: This study provides a detailed plan for a CDS tool that uses the latest AI technology designed to aid nurses in their day‐to‐day HHC service. Our proposed CDS tool includes an alert system that serves as a guard rail to prevent ED visits and hospitalizations. This tool can potentially improve how nurses make decisions and improve patient outcomes by providing early warnings about ED visits and hospitalizations.

Objectives: As artificial intelligence evolves, integrating speech processing into home healthcare (HHC) workflows is increasingly feasible. Audio-recorded communications enhance risk identification models, with automatic speech recognition (ASR) systems as a key component. This study evaluates the transcription accuracy and equity of 4 ASR systems—Amazon Web Services (AWS) General, AWS Medical, Whisper, and Wave2Vec—in transcribing patient-nurse communication in US HHC, focusing on their ability in accurate transcription of speech from Black and White English-speaking patients. Materials and Methods: We analyzed audio recordings of patient-nurse encounters from 35 patients (16 Black and 19 White) in a New York City-based HHC service. Overall, 860 utterances were available for study, including 475 drawn from Black patients and 385 from White patients. Automatic speech recognition performance was measured using word error rate (WER), benchmarked against a manual gold standard. Disparities were assessed by comparing ASR performance across racial groups using the linguistic inquiry and word count (LIWC) tool, focusing on 10 linguistic dimensions, as well as specific speech elements including repetition, filler words, and proper nouns (medical and nonmedical terms). Results: The average age of participants was 67.8 years (SD = 14.4). Communication lasted an average of 15 minutes (range: 11-21 minutes) with a median of 1186 words per patient. Of 860 total utterances, 475 were from Black patients and 385 from White patients. Amazon Web Services General had the highest accuracy, with a median WER of 39%. However, all systems showed reduced accuracy for Black patients, with significant discrepancies in LIWC dimensions such as “Affect,” “Social,” and “Drives.” Amazon Web Services Medical performed best for medical terms, though all systems have difficulties with filler words, repetition, and nonmedical terms, with AWS General showing the lowest error rates at 65%, 64%, and 53%, respectively. Discussion: While AWS systems demonstrated superior accuracy, significant disparities by race highlight the need for more diverse training datasets and improved dialect sensitivity. Addressing these disparities is critical for ensuring equitable ASR performance in HHC settings and enhancing risk prediction models through audio-recorded communication.

Computational drug repositioning, through predicting drug-disease associations (DDA), offers significant potential for discovering new drug indications. Current methods incorporate graph neural networks (GNN) on drug-disease heterogeneous networks to predict DDAs, achieving notable performances compared to traditional machine learning and matrix factorization approaches. However, these methods depend heavily on network topology, hampered by incomplete and noisy network data, and overlook the wealth of biomedical knowledge available. Correspondingly, large language models (LLMs) excel in graph search and relational reasoning, which can possibly enhance the integration of comprehensive biomedical knowledge into drug and disease profiles. In this study, we first investigate the contribution of LLM-inferred knowledge representation in drug repositioning and DDA prediction. A zero-shot prompting template was designed for LLM to extract high-quality knowledge descriptions for drug and disease entities, followed by embedding generation from language models to transform the discrete text to continual numerical representation. Then, we proposed LLM-DDA with three different model architectures (LLM-DDA_{Node Feat}, LLM-DDA_{Dual GNN}, LLM-DDA_GNN-AE) to investigate the best fusion mode for LLM-based embeddings. Extensive experiments on four DDA benchmarks show that, LLM-DDA_GNN-AE achieved the optimal performance compared to 11 baselines with the overall relative improvement in AUPR of 23.22%, F1-Score of 17.20%, and precision of 25.35%. Meanwhile, selected case studies of involving Prednisone and Allergic Rhinitis highlighted the model’s capability to identify reliable DDAs and knowledge descriptions, supported by existing literature. This study showcases the utility of LLMs in drug repositioning with its generality and applicability in other biomedical relation prediction tasks. Graphical abstract