A Comprehensive Survey of Multimodal Foundation Model Evaluation: A Hierarchical Perspective and Extensive Applications
Authors:
- Ye YuanYYYe Yuan
- Junyu LuoJLJunyu Luo
- Guancheng WanGWGuancheng Wan
- Jinsheng HuangJHJinsheng Huang
- Chengwu LiuCLChengwu Liu
- Junwei YangJYJunwei Yang
- Yifang QinYQYifang Qin
- Zhiping XiaoZXZhiping Xiao
- Qingqing LongQLQingqing Long
- Meng XiaoMeng Xiao
- Yiqiao JinYJYiqiao Jin
- Jianhong TuJTJianhong Tu
Preprint
- 24 August 2026
- preprint
- Published by Elsevier in SSRN Electronic Journal
Abstract
Multimodal foundation models increasingly support querying, interpreting, and acting on heterogeneous data and knowledge sources, but their evaluation remains fragmented. Existing benchmarks are scattered across modalities, tasks, data sources, metrics, and deployment assumptions, making it difficult to judge whether a model can support reliable decisions in data- and knowledge-intensive settings. This survey reviews multimodal foundation model evaluation from a multi-source information fusion perspective. We organize representative datasets, metrics, and protocols into a hierarchical, three-layer evaluation pathway: fundamental competencies, higher-order intelligence, and real-world applications. The first two layers group capabilities, whereas the third examines how those capabilities are evaluated under deployment and domain constraints rather than asserting a universal difficulty ordering. We use this taxonomy to connect evaluations of perception and alignment with those of multimodal reasoning, offline temporal comprehension, online streaming-input understanding, decision making, and planning. We also examine deployment-oriented assessment in healthcare, biology, industry, and society. We characterize benchmark coverage, reported performance, adoption signals, and evaluation design choices. Across these resources, recurring risks include data contamination, insufficient process evidence, weak robustness testing, and limited domain validation. Based on this synthesis, we provide a reporting checklist for building and interpreting evaluation resources. An accompanying repository tracks new benchmarks and related papers.Keywords
This publication has 63 references indexed in Scilit:
- RxnBench: A Multimodal Benchmark for Evaluating Large Language Models on Chemical Reaction Understanding from the Scientific LiteratureJournal of Chemical Information and Modeling, 2026
- EgoPlan-Bench: Benchmarking Multimodal Large Language Models for Human-Level PlanningInternational Journal of Computer Vision, 2026
- Multitask benchmarking of single-cell multimodal omics integration methodsNature Methods, 2025
- scCompass: An Integrated Multi‐Species scRNA‐seq Database for AI‐ReadyAdvanced Science, 2025
- Multimodal generative AI for medical image interpretationNature, 2025
- GeneCompass: deciphering universal gene regulatory mechanisms with a knowledge-informed cross-species foundation modelCell Research, 2024
- A Diagram is Worth a Dozen Images2016
- Multi-omic data analysis using GalaxyNature Biotechnology, 2015
- Multisensor data fusion: A review of the state-of-the-artInformation Fusion, 2013
- Multimodal fusion for multimedia analysis: a surveyMultimedia Systems, 2010