Ethan Goh

Clinical artificial intelligence (AI) has advanced rapidly, with frontier large language models now matching or exceeding physician performance on simulated diagnostic reasoning and clinical decision-support tasks. Yet adoption has outpaced the evidence base: fewer than 5% of cleared U.S. Food and Drug Administration (FDA) AI/machine learning (ML) devices have undergone peer-reviewed evaluation and prospective randomised trials remain scarce. This narrative review synthesises evidence from ARISE’s State of Clinical AI Report 2026 across six domains: model performance and human benchmarking, evaluation methodology and benchmark validity, foundational methods including multimodal and multiagent systems, clinical workflow integration, patient-facing applications and applied domain-specific AI. We conducted a targeted literature search of PubMed, medRxiv and arXiv supplemented by expert nomination, including empirical studies of AI in clinical contexts. Across domains, frontier models demonstrate strong clinical reasoning capabilities but exhibit persistent failures in uncertainty calibration, metacognition and robustness to distributional shift. Standard benchmarks have saturated, prompting development of multidimensional evaluation frameworks that assess safety, real-world workflows and agentic capabilities. New trends in foundational methods include converting medical data into tokens and developing multiagent, multimodal systems. In clinical workflows, early prospective trials in clinical decision support and diagnostic imaging show promising results, though human-AI collaboration remains suboptimal and risks of automation bias and clinician deskilling are emerging. In tandem, patient-facing AI is progressing with more personalised health assistance; with this comes the risk of patient overtrust, raising the bar for guardrails. The most credible translational progress occurs on narrowly defined tasks with clear endpoints, while broader clinical autonomy awaits advances in evaluation and human-AI interaction.

' Context engineering is the delicate art and science of filling the context window with just the right information for the next step.” Andrej Karpathy

More than 65 years ago, complex clinical diagnostic reasoning cases were introduced as the gold standard for the evaluation of expert medical computing systems, a standard that has held ever since. In this study, we report the results of a physician evaluation of a large language model (LLM) on challenging clinical cases across five experiments with a baseline of hundreds of physicians. We then report a real-world study comparing human expert and artificial intelligence (AI) second opinions in randomly selected patients in the emergency room of a major tertiary academic medical center. In all experiments, the LLM outperformed physician baselines and displayed continued improvement from prior generations of AI clinical decision support. Our study suggests that LLMs have eclipsed most benchmarks of clinical reasoning, motivating the urgent need for prospective trials.

Clinical evaluation of large language models (LLMs) currently relies on static datasets and isolated scenarios that fail to capture the cascading effects of healthcare decisions. We propose the Clinical Environment Simulator (CES), a framework that evaluates clinical LLMs within digital hospital environments where every decision dynamically alters future states. The CES would use a parallel simulation architecture: a 'hospital engine' that tracks bed availability, staff workloads and equipment status in real time, and a 'patient engine' that simulates disease progression and treatment responses based on LLM interventions. Unlike current benchmarks, the CES framework requires clinical LLMs to execute decisions through realistic electronic health record interfaces, while managing trade-offs between individual patient optimization and system-wide efficiency. The CES enables three critical evaluations absent from current benchmarks: temporal reasoning under evolving constraints, where delayed diagnostics can lead to patient deterioration; resource-aware decision-making, where aggressive workups for one patient may exhaust capacity needed by others; and operational resilience, through adversarial testing with simultaneous emergencies and system failures. By scoring LLM performance on both clinical outcomes and operational metrics, the CES represents a shift toward evaluating clinical LLMs as a dynamic and integrated component of healthcare delivery systems.

While large language models (LLMs) achieve near-perfect scores on medical licensing exams, these evaluations inadequately reflect the complexity and diversity of real-world clinical practice. Here we introduce MedHELM, an extensible evaluation framework with three contributions. First, a clinician-validated taxonomy organizing medical AI applications into five categories that mirror real clinical tasks—clinical decision support (diagnostic decisions, treatment planning), clinical note generation (visit documentation, procedure reports), patient communication (education materials, care instructions), medical research (literature analysis, clinical data analysis) and administration (scheduling, workflow coordination). These encompass 22 subcategories and 121 specific tasks reflecting daily medical practice. Second, a comprehensive benchmark suite of 37 evaluations covering all subcategories. Third, systematic comparison of nine frontier LLMs—Claude 3.5 Sonnet, Claude 3.7 Sonnet, DeepSeek R1, Gemini 1.5 Pro, Gemini 2.0 Flash, GPT-4o, GPT-4o mini, Llama 3.3 and o3-mini—using an automated LLM-jury evaluation method. Our LLM-jury uses multiple AI evaluators to assess model outputs against expert-defined criteria. Advanced reasoning models (DeepSeek R1, o3-mini) demonstrated superior performance with win rates of 66%, although Claude 3.5 Sonnet achieved comparable results at 15% lower computational cost. These results not only highlight current model capabilities but also demonstrate how MedHELM could enable evidence-based selection of medical AI systems for healthcare applications.