InSight-o3: Empowering Multimodal Foundation Models with Generalized Visual Search
Authors:
- Tiezheng YuTYTiezheng Yu
- Nevin L. ZhangNZNevin L. Zhang
Preprint
Abstract
The ability for AI agents to "think with images" requires a sophisticated blend of reasoning and perception. However, current open multimodal agents still largely fall short on the reasoning aspect crucial for real-world tasks like analyzing documents with dense charts/diagrams and navigating maps. To address this gap, we introduce O3-Bench, a new benchmark designed to evaluate multimodal reasoning with interleaved attention to visual details. O3-Bench features challenging problems that require agents to piece together subtle visual information from distinct image areas through multi-step reasoning. The problems are highly challenging even for frontier systems like OpenAI o3, which only obtains 40.8% accuracy on O3-Bench. To make progress, we propose InSight-o3, a multi-agent framework consisting of a visual reasoning agent (vReasoner) and a visual search agent (vSearcher) for which we introduce the task of generalized visual search -- locating relational, fuzzy, or conceptual regions described in free-form language, beyond just simple objects or figures in natural images. We then present a multimodal LLM purpose-trained for this task via reinforcement learning. As a plug-and-play agent, our vSearcher empowers frontier multimodal models (as vReasoners), significantly improving their performance on a wide range of benchmarks. This marks a concrete step towards powerful o3-like open systems. Our code and dataset can be found at https://github.com/m-Just/InSight-o3 .Keywords
All Related Versions
This publication has 11 references indexed in Scilit:
- HybridFlow: A Flexible and Efficient RLHF Framework2025
- ZoomEye: Enhancing Multimodal LLMs with Human-Like Zooming Capabilities through Tree-Based Image Exploration2025
- MMMU-Pro: A More Robust Multi-discipline Multimodal Understanding Benchmark2025
- Measuring Multimodal Mathematical Reasoning with MATH-Vision Dataset2024
- Direct Preference Optimization: Your Language Model is Secretly a Reward Model2023
- CAMEL: Communicative Agents for "Mind" Exploration of Large Language Model Society2023
- HuggingGPT: Solving AI Tasks with ChatGPT and its Friends in Hugging Face2023
- Evaluating Object Hallucination in Large Vision-Language Models2023
- CODA: A Real-World Road Corner Case Dataset for Object Detection in Autonomous Driving2022
- FAIR1M: A benchmark dataset for fine-grained object recognition in high-resolution remote sensing imageryISPRS Journal of Photogrammetry and Remote Sensing, 2022