A Comprehensive Survey of Multimodal Foundation Model Evaluation: A Hierarchical Perspective and Extensive Applications

Authors:

Abstract
Multimodal foundation models increasingly support querying, interpreting, and acting on heterogeneous data and knowledge sources, but their evaluation remains fragmented. Existing benchmarks are scattered across modalities, tasks, data sources, metrics, and deployment assumptions, making it difficult to judge whether a model can support reliable decisions in data- and knowledge-intensive settings. This survey reviews multimodal foundation model evaluation from a multi-source information fusion perspective. We organize representative datasets, metrics, and protocols into a hierarchical, three-layer evaluation pathway: fundamental competencies, higher-order intelligence, and real-world applications. The first two layers group capabilities, whereas the third examines how those capabilities are evaluated under deployment and domain constraints rather than asserting a universal difficulty ordering. We use this taxonomy to connect evaluations of perception and alignment with those of multimodal reasoning, offline temporal comprehension, online streaming-input understanding, decision making, and planning. We also examine deployment-oriented assessment in healthcare, biology, industry, and society. We characterize benchmark coverage, reported performance, adoption signals, and evaluation design choices. Across these resources, recurring risks include data contamination, insufficient process evidence, weak robustness testing, and limited domain validation. Based on this synthesis, we provide a reporting checklist for building and interpreting evaluation resources. An accompanying repository tracks new benchmarks and related papers.

This publication has 63 references indexed in Scilit: