MediConfusion: Can you trust your AI radiologist? Probing the reliability of multimodal medical foundation models
Preprint
Abstract
Multimodal Large Language Models (MLLMs) have tremendous potential to improve the accuracy, availability, and cost-effectiveness of healthcare by providing automated solutions or serving as aids to medical professionals. Despite promising first steps in developing medical MLLMs in the past few years, their capabilities and limitations are not well-understood. Recently, many benchmark datasets have been proposed that test the general medical knowledge of such models across a variety of medical areas. However, the systematic failure modes and vulnerabilities of such models are severely underexplored with most medical benchmarks failing to expose the shortcomings of existing models in this safety-critical domain. In this paper, we introduce MediConfusion, a challenging medical Visual Question Answering (VQA) benchmark dataset, that probes the failure modes of medical MLLMs from a vision perspective. We reveal that state-of-the-art models are easily confused by image pairs that are otherwise visually dissimilar and clearly distinct for medical experts. Strikingly, all available models (open-source or proprietary) achieve performance below random guessing on MediConfusion, raising serious concerns about the reliability of existing medical MLLMs for healthcare deployment. We also extract common patterns of model failure that may help the design of a new generation of more trustworthy and reliable MLLMs in healthcare.Keywords
All Related Versions
This publication has 9 references indexed in Scilit:
- Towards generalist foundation model for radiology by leveraging web-scale 2D&3D medical dataNature Communications, 2025
- Beyond the Hype: A Dispassionate Look at Vision–Language Models in Medical ScenarioIEEE Transactions on Neural Networks and Learning Systems, 2025
- Large language models for robotics: Opportunities, challenges, and perspectivesJournal of Automation and Intelligence, 2025
- Asclepius: A Spectrum Evaluation Benchmark for Medical Multi-Modal Large Language Models2025
- LVLM-EHub: A Comprehensive Evaluation Benchmark for Large Vision-Language ModelsIEEE Transactions on Pattern Analysis and Machine Intelligence, 2024
- Improved Baselines with Visual Instruction Tuning2024
- GMAI-MMBench: A Comprehensive Multimodal Evaluation Benchmark Towards General Medical AI2024
- CARES: A Comprehensive Benchmark of Trustworthiness in Medical Vision Language Models2024
- A dataset of clinically generated visual questions and answers about radiology imagesScientific Data, 2018