Incorporating LLM Versus LLM Into Multimodal Chain-of-Thought for Fine-Grained Evidence Generation
Open Access
- 26 November 2025
- journal article
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Access
- Vol. 13, 202143-202170
Abstract
Multimodal Chain-of-Thought (MCoT) has become an effective strategy for enhancing multimodal large-language models (MLLMs) by breaking down complex tasks into sequential reasoning steps. Despite its interpretability benefits, MCoT often encounters difficulties with fine-grained semantic grounding, particularly when reasoning involves small objects, subtle attributes, or visually complex scenes that can lead to inaccuracies. Existing attempts to address these issues primarily fall into two categories: fine-tuning, which depends on large annotated datasets and costly parameter updates; and in-context learning (ICL), which achieves few-shot or zero-shot reasoning without model modification. Although ICL provides flexibility and adaptability, it is prone to semantic drift caused by an unstable prompt quality. To overcome these limitations, this study presents an entity-level evidence generation and verification framework using the ICL paradigm. This approach first produces MCoT from multimodal inputs, followed by extraction of key entities with enriched evidential descriptions. These entities were then cross-validated through adversarial checks using multiple MLLMs, and the verified evidence was integrated back into the reasoning chain. Experiments demonstrated consistent performance gains: on ScienceQA, the accuracy improved from 82.39% to 86.04%(+3.65%) with GPT-3.5, 84.96% to 89.37%(+4.41%) with Gemini; on MathVista, the accuracy increased from 43.1% to 43.6%(+0.50%) with GPT-3.5, and from 44.7% to 45.6%(+0.90%) with Gemini. These results establish newstate-of-the-art baselines and confirm the robustness and generalizability of the entity-level verification for multimodal reasoning.Keywords
This publication has 23 references indexed in Scilit:
- Exploring Embodied Multimodal Large Models: Development, datasets, and future directionsInformation Fusion, 2025
- Approaching the Quantum Limit of Energy Resolution in Animal MagnetoreceptionPRX Life, 2025
- LTNER: Large Language Model Tagging for Named Entity Recognition with Contextualized Entity Marking2024
- Cantor: Inspiring Multimodal Chain-of-Thought of MLLM2024
- MMMU: A Massive Multi-Discipline Multimodal Understanding and Reasoning Benchmark for Expert AGI2024
- Graph of Thoughts: Solving Elaborate Problems with Large Language ModelsProceedings of the AAAI Conference on Artificial Intelligence, 2024
- KAM-CoT: Knowledge Augmented Multimodal Chain-of-Thoughts ReasoningProceedings of the AAAI Conference on Artificial Intelligence, 2024
- Delving into Multimodal Prompting for Fine-Grained Visual ClassificationProceedings of the AAAI Conference on Artificial Intelligence, 2024
- T-SciQ: Teaching Multimodal Chain-of-Thought Reasoning via Large Language Model Signals for Science Question AnsweringProceedings of the AAAI Conference on Artificial Intelligence, 2024
- Comparison study of dispatching rules and heuristics for online scheduling of single machine scheduling problem with predicted release time jobsExpert Systems with Applications, 2023