AgentClinic: a multimodal benchmark for tool-using clinical AI agents
Top Cited Papers
Open Access
- 27 April 2026
- journal article
- Published by Springer Nature in npj Digital Medicine
- Vol. 9
Abstract
Evaluating large language models (LLM) in clinical scenarios is crucial to assessing their potential clinical utility. Existing benchmarks rely heavily on static question-answering, which does not accurately depict the complex, sequential nature of clinical decision-making. Here, we introduce AgentClinic, a multimodal agent benchmark for evaluating LLMs in simulated clinical environments that include patient interactions, multimodal data collection under incomplete information, and the usage of various tools, resulting in an in-depth evaluation across nine medical specialties and seven languages. We find that solving MedQA problems in the sequential decision-making format of AgentClinic is considerably more challenging, resulting in diagnostic accuracies that can drop to below a tenth of the original accuracy. Overall, we observe that agents sourced from Claude-3.5 outperform other LLM backbones in most settings. Nevertheless, we see stark differences in the LLMs’ ability to make use of tools, such as experiential learning, adaptive retrieval, and reflection cycles. Strikingly, Llama-3 shows up to 92% relative improvements with the notebook tool that allows for writing and editing notes that persist across cases. To further scrutinize our clinical simulations, we leverage real-world electronic health records, perform a clinical reader study, perturb agents with biases, and explore patient-centric metrics that this interactive environment enables.Keywords
Funding Information
- National Science Foundation,United States (DGE 2139757)
- National Science Foundation,United States (DGE 2139757)
This publication has 19 references indexed in Scilit:
- Large language models encode clinical knowledgeNature, 2023
- MIMIC-IV, a freely accessible electronic health record datasetScientific Data, 2023
- Tackling Implicit Bias in Health CareNew England Journal of Medicine, 2022
- Implicit bias in healthcare: clinical practice, research and decision makingFuture Healthcare Journal, 2021
- Bias in MedicineJACC: Basic To Translational Science, 2021
- Waiting Time as an Indicator for Health Services Under Strain: A Narrative ReviewINQUIRY: The Journal of Health Care Organization, Provision, and Financing, 2020
- Implicit bias in healthcare professionals: a systematic reviewBMC Medical Ethics, 2017
- Cognitive Biases and Heuristics in Medical Decision MakingMedical Decision Making, 2014
- Use of Diagnostic Imaging Studies and Associated Radiation Exposure for Patients Enrolled in Large Integrated Health Care Systems, 1996-2010JAMA, 2012
- Analysis of questions asked by family doctors regarding patient careBMJ, 1999