Q-MoE: Connector for MLLMs with Text-Driven Routing
Open Access
- 28 October 2024
- conference paper
- Published by Association for Computing Machinery (ACM)
- p. 817-825
Abstract
Multimodal Large Language Models (MLLMs) have showcased remarkable advances in handling various vision-language tasks. These models typically consist of a Large Language Model (LLM), a vision encoder and a connector structure, which is used to bridge the modality gap between vision and language. It is challenging for the connector to filter the right visual information for LLM according to the task in hand. Most of previous connectors, such as light-weight projection and Q-former, treat visual information for diverse tasks uniformly, therefore lacking task-specific visual information extraction capabilities. To address the issue, this paper proposes Q-MoE, a query-based connector with Mixture-of-Experts (MoE) to extract task-specific information with text-driven routing. Furthermore, an optimal path based training strategy is proposed to find an optimal expert combination. Extensive experiments on two popular open-source LLMs and several different visual-language tasks demonstrate the effectiveness of the Q-MoE connecter.Keywords
This publication has 10 references indexed in Scilit:
- Visual Spatial ReasoningTransactions of the Association for Computational Linguistics, 2023
- A-OKVQA: A Benchmark for Visual Question Answering Using World Knowledge2022
- TextCaps: A Dataset for Image Captioning with Reading Comprehension2020
- nocaps: novel object captioning at scale2019
- GQA: A New Dataset for Real-World Visual Reasoning and Compositional Question Answering2019
- OK-VQA: A Visual Question Answering Benchmark Requiring External Knowledge2019
- VizWiz Grand Challenge: Answering Visual Questions from Blind People2018
- Making the V in VQA Matter: Elevating the Role of Image Understanding in Visual Question Answering2017
- ReferItGame: Referring to Objects in Photographs of Natural Scenes2014
- Adaptive Mixtures of Local ExpertsNeural Computation, 1991