Mixture-of-Visual-Thoughts: Exploring Context-Adaptive Reasoning Mode Selection for General Visual Reasoning
Preprint
Abstract
Current visual reasoning methods mainly focus on exploring specific reasoning modes. Although improvements can be achieved in particular domains, they struggle to develop general reasoning capabilities. Inspired by this, we propose a novel adaptive reasoning paradigm, Mixture-of-Visual-Thoughts (MoVT), which unifies different reasoning modes within a single model and guides it to select the appropriate mode based on context. To achieve this, we introduce AdaVaR, a two-stage Adaptive Visual Reasoning learning framework: different modes are unified and learned during the supervised cold-start stage, and the mode selection capability is induced via an RL process with a carefully designed AdaGRPO algorithm. Extensive experiments show that AdaVaR effectively guides the model to learn and differentiate multiple modes and perform context-adaptive mode selection, achieving consistent improvement across various scenarios, highlighting MoVT as an effective solution for building general visual reasoning models.Keywords
All Related Versions
This publication has 11 references indexed in Scilit:
- ZoomEye: Enhancing Multimodal LLMs with Human-Like Zooming Capabilities through Tree-Based Image Exploration2025
- LlamaV-o1: Rethinking Step-by-step Visual Reasoning in LLMs2025
- We-Math: Does Your Large Multimodal Model Achieve Human-like Mathematical Reasoning?2025
- VoCoT: Unleashing Visually Grounded Multi-Step Reasoning in Large Multi-Modal Models2025
- MATHVERSE: Does Your Multi-modal LLM Truly See the Diagrams in Visual Math Problems?2024
- V*: Guided Visual Search as a Core Mechanism in Multimodal LLMs2024
- Are We on the Right Way for Evaluating Large Vision-Language Models?2024
- Reflexion: language agents with verbal reinforcement learning2023
- Evaluating Object Hallucination in Large Vision-Language Models2023
- HyperTree Proof Search for Neural Theorem Proving2022