Mixture-of-Retrieval Experts for Reasoning-Guided Multimodal Knowledge Exploitation
Open Access
- 19 July 2026
- conference paper
- Published by Association for Computing Machinery (ACM)
- p. 1440-1450
Abstract
Multimodal Retrieval-Augmented Generation (MRAG) has shown promise in mitigating hallucinations in Multimodal Large Language Models (MLLMs) by incorporating external knowledge. However, existing methods typically adhere to rigid retrieval paradigms by mimicking fixed retrieval trajectories and thus fail to fully exploit the knowledge of different retrieval experts through dynamic interaction based on the model's knowledge needs or evolving reasoning states. To overcome this limitation, we introduce Mixture-of-Retrieval Experts (MoRE), a novel framework that enables MLLMs to collaboratively interact with diverse retrieval experts for more effective knowledge exploitation. Specifically, MoRE learns to dynamically determine which expert to engage with, conditioned on the evolving reasoning state. To effectively train this capability, we propose Stepwise Group Relative Policy Optimization (Step-GRPO), which goes beyond sparse outcome-based supervision by encouraging MLLMs to interact with multiple retrieval experts and synthesize fine-grained rewards, thereby teaching the MLLM to fully coordinate all experts when answering a given query. Experimental results on diverse open-domain QA benchmarks demonstrate the effectiveness of MoRE, achieving average performance gains of over 7% compared to competitive baselines. Notably, MoRE exhibits strong adaptability by dynamically coordinating heterogeneous experts to precisely locate relevant information, validating its capability for robust, reasoning-driven expert collaboration. All codes and data are released on https://github.com/OpenBMB/MoRE.Keywords
All Related Versions
Funding Information
- National Natural Science Foundation of China (62576082)
- National Natural Science Foundation of China (62461146205)
This publication has 6 references indexed in Scilit:
- R1-Onevision: Advancing Generalized Multimodal Reasoning Through Cross-Modal Formalization2025
- Wiki-LLaVA: Hierarchical Retrieval-Augmented Generation for Multimodal LLMs2024
- RULE: Reliable Multimodal RAG for Factuality in Medical Vision Language Models2024
- Direct Preference Optimization: Your Language Model is Secretly a Reward Model2023
- Augmenting Zero-Shot Dense Retrievers with Plug-in Mixture-of-Memories2023
- WebQA: Multihop and Multimodal QA2022