AI Agents in Clinical Medicine: A Systematic Review
Top Cited Papers
Preprint
Abstract
Background: AI agents built on large language models (LLMs) can plan tasks, use external tools, and coordinate with other agents. Unlike standard LLMs, agents can execute multi-step processes, access real-time clinical information, and integrate multiple data sources. There has been interest in using such agents for clinical and administrative tasks, however, there is limited knowledge on their performance and whether multi-agent systems function better than a single agent for healthcare tasks. Purpose: To evaluate the performance of AI agents in healthcare, compare AI agent systems vs. standard LLMs and catalog the tools used for task completion Data Sources: PubMed, Web of Science, and Scopus from October 1, 2022, through August 5, 2025. Study Selection: Peer-reviewed studies implementing AI agents for clinical tasks with quantitative performance comparisons. Data Extraction: Two reviewers (A.G., M.O.) independently extracted data on architectures, performance metrics, and clinical applications. Discrepancies were resolved by discussion, with a third reviewer (E.K.) consulted when consensus could not be reached. Data Synthesis: Twenty studies met inclusion criteria. Across studies, all agent systems outperformed their baseline LLMs in accuracy performance. Improvements ranged from small gains to increases of over 60 percentage points, with a median improvement of 53 percentage points in single-agent tool-calling studies. These systems were particularly effective for discrete tasks such as medication dosing and evidence retrieval. Multi-agent systems showed optimal performance with up to 5 agents, and their effectiveness was particularly pronounced when dealing with highly complex tasks. The highest performance boost occurred when the complexity of the AI agent framework aligned with that of the task. Limitations: Heterogeneous outcomes precluded quantitative meta-analysis. Several studies relied on synthetic data, limiting generalizability. Conclusions: AI agents consistently improve clinical task performance of Base-LLMs when architecture matches task complexity. Our analysis indicates a step-change over base-LLMs, with AI agents opening previously inaccessible domains. Future efforts should be based on prospective, multi-center trials using real-world data to determine safety, task matched and cost-effectiveness. Primary Funding Source: This work was supported in part through the computational and data resources and staff expertise provided by Scientific Computing and Data at the Icahn School of Medicine at Mount Sinai and supported by the Clinical and Translational Science Awards (CTSA) grant UL1TR004419 from the National Center for Advancing Translational Sciences. Research reported in this publication was also supported by the Office of Research Infrastructure of the National Institutes of Health under award number S10OD026880 and S10OD030463. The content is solely the responsibility of the authors and does not necessarily represent the official views of the National Institutes of Health. Registration: PROSPERO CRD420251120318Keywords
This publication has 74 references indexed in Scilit:
- AI-Based EMG Reporting: A Randomized Controlled TrialJournal of Neurology, 2025
- We need a new ethics for a world of AI agentsNature, 2025
- Refining LLMs outputs with iterative consensus ensemble (ICE)Computers in Biology and Medicine, 2025
- Agent-guided AI-powered interpretation and reporting of nerve conduction studies and EMG (INSPIRE)Clinical Neurophysiology, 2025
- LITERAS: Biomedical literature review and citation retrieval agentsComputers in Biology and Medicine, 2025
- The rise of agentic AI teammates in medicineThe Lancet, 2025
- Custom Large Language Models Improve Accuracy: Comparing Retrieval Augmented Generation and Artificial Intelligence Agents to Noncustom Models for Evidence‐Based MedicineArthroscopy, 2024
- Large Language Model Influence on Diagnostic ReasoningJAMA Network Open, 2024
- Measuring Documentation Burden in HealthcareJournal of General Internal Medicine, 2024
- Medical Documentation Burden Among US Office-Based Physicians in 2019JAMA Internal Medicine, 2022