Fateme Nateghi Haredasht

Background: Health systems answer most questions by having expert analysts hand-write queries against a complex electronic health record data warehouse, a slow, resource-intensive process. Whether an autonomous coding agent can do this accurately is unknown. Methods: In a single-center quality-improvement evaluation, we posed ten questions about a common pediatric infection, acute otitis media. The questions went to analysts, whose adjudicated answers were the reference, and to an autonomous coding agent (OpenAI Codex), which wrote and ran read-only queries on a full copy of the production data warehouse (Epic Caboodle). We ran the agent under four conditions: an autonomous baseline (each question answered three times at two reasoning-effort settings), a variant in which it listed its assumptions, interactive analyst feedback, and reuse of a corrected definition across related questions. Outcomes were accuracy, patient-level agreement (F1), reproducibility, and efficiency (time and tokens). Results: Working autonomously, the agent wrote valid queries and never fabricated data, but rarely produced the exact answer. At medium effort, it came within 5% of the reference on 27 of 30 runs but exact on only 10. Higher effort produced no improvement. Reproducibility was the greater weakness, with the three runs returning an identical answer on only 3 of 10 questions. When a run matched the reference, it had found the same patients (F1 = 1.00); the exception was an over-counted procedure (F1 = 0.72). With analyst feedback on four questions, it answered two exactly, and reusing the corrected definition on related questions restored reproducibility and accuracy. Conclusions: A general-purpose coding agent matched an adjudicated analyst reference on most routine questions but not reproducibly, because key definitions depended on warehouse knowledge that the data dictionary omits. Supplying that knowledge, by prompt or feedback, restored reproducibility. Under expert analyst supervision, the agent is already a capable drafting aid and a promising step toward broader hospital analytic support.

Background: The 2024 blood culture bottle shortage brought diagnostic resource allocation to the forefront, reflecting persistent, foundational challenges with low-value testing and empiric treatment approaches under clinical uncertainty. Objective: To determine whether a machine learning approach using electronic medical record data can predict bacteremia more effectively than existing systems and practices to guide diagnostic testing and empiric treatment strategies. Methods: In a retrospective cohort of 101,812 adult emergency department encounters (2015-2025), we first established an idealized cognitive baseline by evaluating physician and generative AI (GPT-5) application of the professional society-endorsed Fabre framework on a validation subset. We then trained an XGBoost model (Cultryx) on the full cohort to predict bacteremia, benchmarking its performance against real-world clinical heuristics (SIRS, Shapiro Rule). Results: For the idealized baseline, physicians applying the Fabre framework achieved 95.7% sensitivity, but GPT-5 automation failed to replicate this standard (71.6% sensitivity). In real-world benchmarking, Cultryx outperformed all clinical heuristics (AUROC 0.810). SIRS lacked specificity (41.2%), driving diagnostic overuse, while the Shapiro Rule lacked sensitivity (70.2%), missing ~30% of bacteremia cases. In contrast, when calibrated to a strict 95% sensitivity target, Cultryx achieved the highest culture volume deferral rate (26.2%, deferring ~ 15,872 bottles with predicted negative results) while maintaining a 98.9% negative predictive value. Cultryx ^score , a simplified bedside tool, retained a 20.8% deferral rate. Conclusions: Machine learning provides a superior, data-driven alternative to mainstream clinical heuristics for predicting bacteremia. By maximizing culture deferment without compromising pathogen detection, Cultryx can conserve diagnostic resources, reduce unnecessary empiric antibiotic exposure, and systematically elevate patient safety. Summary: Cultryx, a machine learning model for blood culture stewardship, outperforms standard clinical heuristics in predicting bacteremia. This approach could reduce culture utilization by over 26% while preserving pathogen detection, conserving diagnostic resources, reducing unnecessary antibiotic exposure, and elevating patient safety.

While large language models (LLMs) achieve near-perfect scores on medical licensing exams, these evaluations inadequately reflect the complexity and diversity of real-world clinical practice. Here we introduce MedHELM, an extensible evaluation framework with three contributions. First, a clinician-validated taxonomy organizing medical AI applications into five categories that mirror real clinical tasks—clinical decision support (diagnostic decisions, treatment planning), clinical note generation (visit documentation, procedure reports), patient communication (education materials, care instructions), medical research (literature analysis, clinical data analysis) and administration (scheduling, workflow coordination). These encompass 22 subcategories and 121 specific tasks reflecting daily medical practice. Second, a comprehensive benchmark suite of 37 evaluations covering all subcategories. Third, systematic comparison of nine frontier LLMs—Claude 3.5 Sonnet, Claude 3.7 Sonnet, DeepSeek R1, Gemini 1.5 Pro, Gemini 2.0 Flash, GPT-4o, GPT-4o mini, Llama 3.3 and o3-mini—using an automated LLM-jury evaluation method. Our LLM-jury uses multiple AI evaluators to assess model outputs against expert-defined criteria. Advanced reasoning models (DeepSeek R1, o3-mini) demonstrated superior performance with win rates of 66%, although Claude 3.5 Sonnet achieved comparable results at 15% lower computational cost. These results not only highlight current model capabilities but also demonstrate how MedHELM could enable evidence-based selection of medical AI systems for healthcare applications.

This study evaluates the capacity of large language models (LLMs) to generate structured clinical consultation templates for electronic consultation. Using 145 expert-crafted templates developed and routinely used by Stanford's eConsult team, we assess frontier models-including o3, GPT-4o, Kimi K2, Claude 4 Sonnet, Llama 3 70B, and Gemini 2.5 Pro-for their ability to produce clinically coherent, concise, and prioritized clinical question schemas. Through a multi-agent pipeline combining prompt optimization, semantic autograding, and prioritization analysis, we show that while models like o3 achieve high comprehensiveness (up to 92.2%), they consistently generate excessively long templates and fail to correctly prioritize the most clinically important questions under length constraints. Performance varies across specialties, with significant degradation in narrative-driven fields such as psychiatry and pain medicine. Our findings demonstrate that LLMs can enhance structured clinical information exchange between physicians, while highlighting the need for more robust evaluation methods that capture a model's ability to prioritize clinically salient information within the time constraints of real-world physician communication. Limitations include reliance on Stanford-specific templates and concordancebased grading, which may not capture all clinically reasonable outputs.

Asynchronous patient-clinician messaging via EHR portals is a growing source of clinician workload, prompting interest in large language models (LLMs) to assist with draft responses. However, LLM outputs may contain clinical inaccuracies, omissions, or tone mismatches, making robust evaluation essential. Our contributions are threefold: (1) we introduce a clinically grounded error ontology comprising 5 domains and 59 granular error codes, developed through inductive coding and expert adjudication; (2) we develop a Retrieval-Augmented Error Checking (RAEC) pipeline that leverages semantically similar historical message-response pairs to improve judgment quality; and (3) we provide a two-stage prompting architecture using DSPy to enable scalable, interpretable, and hierarchical error detection. Our approach assesses the quality of drafts both in isolation and with reference to similar past message-response pairs retrieved from institutional archives. Using a two-stage DSPy pipeline, we compared baseline and reference-enhanced evaluations on over 1,500 patient messages. Retrieval context improved error identification in domains such as clinical completeness and workflow appropriateness. Human validation on 100 messages demonstrated superior agreement (concordance = 50% vs. 33%) and performance (F1 = 0.500 vs. 0.256) of context-enhanced labels vs. baseline, supporting the use of our RAEC pipeline as AI guardrails for patient messaging.

Artificial Intelligence (AI) technologies continue to expand their role in clinical medicine, with large language models (LLMs) and multimodal systems now applied to communication, imaging, and predictive analytics. Advances in generative and retrieval-augmented methods have improved the accuracy and contextual grounding of clinical summaries, patient messaging, and decision support. At the same time, new benchmarks in imaging, vision, and spontaneous speech have underscored both progress and the persistence of unsolved challenges. Predictive modeling efforts highlight causality, longitudinal trajectories, and informative clinical events, while methodological contributions emphasize uncertainty management, abstention, and interpretable causal structures. Finally, frameworks for evaluation and governance address the crucial gap between laboratory performance and real-world deployment.

Asynchronous patient-clinician messaging via EHR portals is a growing source of clinician workload, prompting interest in large language models (LLMs) to assist with draft responses. However, LLM outputs may contain clinical inaccuracies, omissions, or tone mismatches, making robust evaluation essential. Our contributions are threefold: (1) we introduce a clinically grounded error ontology comprising 5 domains and 59 granular error codes, developed through inductive coding and expert adjudication; (2) we develop a retrieval-augmented evaluation pipeline (RAEC) that leverages semantically similar historical message-response pairs to improve judgment quality; and (3) we provide a two-stage prompting architecture using DSPy to enable scalable, interpretable, and hierarchical error detection. Our approach assesses the quality of drafts both in isolation and with reference to similar past message-response pairs retrieved from institutional archives. Using a two-stage DSPy pipeline, we compared baseline and reference-enhanced evaluations on over 1,500 patient messages. Retrieval context improved error identification in domains such as clinical completeness and workflow appropriateness. Human validation on 100 messages demonstrated superior agreement (concordance = 50% vs. 33%) and performance (F1 = 0.500 vs. 0.256) of context-enhanced labels vs. baseline, supporting the use of our RAEC pipeline as AI guardrails for patient messaging.

Objective: Building upon our previous work on predicting treatment retention in medications for opioid use disorder, we aimed to improve 6-month retention prediction in buprenorphine-naloxone (BUP-NAL) therapy by incorporating features derived from large language models (LLMs) applied to unstructured clinical notes. Materials and Methods: We used de-identified electronic health record (EHR) data from Stanford Health Care (STARR) for model development and internal validation, and the NeuroBlu behavioral health database for external validation. Structured features were supplemented with 13 clinical and psychosocial features extracted from free-text notes using the CLinical Entity Augmented Retrieval pipeline, which combines named entity recognition with LLM-based classification to provide contextual interpretation. We trained classification (Logistic Regression, Random Forest, XGBoost) and survival models (CoxPH, Random Survival Forest, Survival XGBoost), evaluated using Receiver Operating Characteristic-Area Under the Curve (ROC-AUC) and C-index. Results: XGBoost achieved the highest classification performance (ROC-AUC = 0.65). Incorporating LLM-derived features improved model performance across all architectures, with the largest gains observed in simpler models such as Logistic Regression. In time-to-event analysis, Random Survival Forest and Survival XGBoost reached the highest C-index (≈0.65). SHapley Additive exPlanations analysis identified LLM-extracted features like Chronic Pain, Liver Disease, and Major Depression as key predictors. We also developed an interactive web tool for real-time clinical use. Discussion: Features extracted using NLP and LLM-assisted methods improved model accuracy and interpretability, revealing valuable psychosocial risks not captured in structured EHRs. Conclusion: Combining structured EHR data with LLM-extracted features moderately improves BUP-NAL retention prediction, enabling personalized risk stratification and advancing AI-driven care for substance use disorders.