GMAI-MMBench: A Comprehensive Multimodal Evaluation Benchmark Towards General Medical AI
Authors:
- Jin YeJYJin Ye
- Tianbin LiTLTianbin Li
Preprint
Abstract
Large Vision-Language Models (LVLMs) are capable of handling diverse data types such as imaging, text, and physiological signals, and can be applied in various fields. In the medical field, LVLMs have a high potential to offer substantial assistance for diagnosis and treatment. Before that, it is crucial to develop benchmarks to evaluate LVLMs' effectiveness in various medical applications. Current benchmarks are often built upon specific academic literature, mainly focusing on a single domain, and lacking varying perceptual granularities. Thus, they face specific challenges, including limited clinical relevance, incomplete evaluations, and insufficient guidance for interactive LVLMs. To address these limitations, we developed the GMAI-MMBench, the most comprehensive general medical AI benchmark with well-categorized data structure and multi-perceptual granularity to date. It is constructed from 285 datasets across 39 medical image modalities, 18 clinical-related tasks, 18 departments, and 4 perceptual granularities in a Visual Question Answering (VQA) format. Additionally, we implemented a lexical tree structure that allows users to customize evaluation tasks, accommodating various assessment needs and substantially supporting medical AI research and applications. We evaluated 50 LVLMs, and the results show that even the advanced GPT-4o only achieves an accuracy of 52%, indicating significant room for improvement. Moreover, we identified five key insufficiencies in current cutting-edge LVLMs that need to be addressed to advance the development of better medical applications. We believe that GMAI-MMBench will stimulate the community to build the next generation of LVLMs toward GMAI. Project Page: https://uni-medical.github.io/GMAI-MMBench.github.io/Keywords
All Related Versions
This publication has 81 references indexed in Scilit:
- ShareGPT4V: Improving Large Multi-modal Models with Better Captions2024
- LLaVA-UHD: An LMM Perceiving Any Aspect Ratio and High-Resolution Images2024
- PathMMU: A Massive Multimodal Expert-Level Benchmark for Understanding and Reasoning in Pathology2024
- The Río Hortega University Hospital Glioblastoma dataset: A comprehensive collection of preoperative, early postoperative and recurrence MRI scans (RHUH-GBM)Data in Brief, 2023
- HaN‐Seg: The head and neck organ‐at‐risk CT and MR segmentation datasetMedical Physics, 2023
- AVT: Multicenter aortic vessel tree CTA dataset collection with ground truth segmentation masksData in Brief, 2022
- Deep learning to segment pelvic bones: large-scale CT datasets and baseline modelsInternational Journal of Computer Assisted Radiology and Surgery, 2021
- Deep-learning-assisted detection and segmentation of rib fractures from CT scans: Development and validation of FracNetEBioMedicine, 2020
- MED-NODE: A computer-assisted melanoma diagnosis system using non-dermoscopic imagesExpert Systems with Applications, 2015
- WM-DOVA maps for accurate polyp highlighting in colonoscopy: Validation vs. saliency maps from physiciansComputerized Medical Imaging and Graphics, 2015