MiCEval: Unveiling Multimodal Chain of Thought's Quality via Image Description and Reasoning Steps
Preprint
Abstract
Multimodal Chain of Thought (MCoT) is a popular prompting strategy for improving the performance of multimodal large language models (MLLMs) across a range of complex reasoning tasks. Despite its popularity, there is a notable absence of automated methods for evaluating the quality of reasoning steps in MCoT. To address this gap, we propose Multimodal Chain-of-Thought Evaluation (MiCEval), a framework designed to assess the correctness of reasoning chains by evaluating the quality of both the description and each reasoning step. The evaluation of the description component focuses on the accuracy of the image descriptions, while the reasoning step evaluates the quality of each step as it is conditionally generated based on the preceding steps. MiCEval is built upon a fine-grained dataset with annotations that rate each step according to correctness, relevance, and informativeness. Extensive experiments on four state-of-the-art MLLMs show that step-wise evaluations using MiCEval align more closely with human judgments compared to existing methods based on cosine similarity or fine-tuning approaches. MiCEval datasets and code can be found in https://github.com/alenai97/MiCEval.Keywords
All Related Versions
This publication has 11 references indexed in Scilit:
- Evaluating Mathematical Reasoning Beyond AccuracyProceedings of the AAAI Conference on Artificial Intelligence, 2025
- Eyes Wide Shut? Exploring the Visual Shortcomings of Multimodal LLMs2024
- ReConcile: Round-Table Conference Improves Reasoning via Consensus among Diverse LLMs2024
- Unveiling the Achilles’ Heel of NLG Evaluators: A Unified Adversarial Framework Driven by Large Language Models2024
- A Chain-of-Thought Is as Strong as Its Weakest Link: A Benchmark for Verifiers of Reasoning Chains2024
- GPTScore: Evaluate as You Desire2024
- SocREval: Large Language Models with the Socratic Method for Reference-free Reasoning Evaluation2024
- Maieutic Prompting: Logically Consistent Reasoning with Recursive Explanations2022
- CLIPScore: A Reference-free Evaluation Metric for Image Captioning2021
- A Study of Automatic Metrics for the Evaluation of Natural Language Explanations2021