OmniBridge: Unified Multimodal Understanding, Generation, and Retrieval via Latent Space Alignment
Authors:
- Teng XiaoTXTeng Xiao
- Zuchao LiZLZuchao Li
- Lefei ZhangLZLefei Zhang
Preprint
Abstract
Recent advances in multimodal large language models (LLMs) have led to significant progress in understanding, generation, and retrieval tasks. However, current solutions often treat these tasks in isolation or require training LLMs from scratch, resulting in high computational costs and limited generalization across modalities. In this work, we present OmniBridge, a unified and modular multimodal framework that supports vision-language understanding, generation, and retrieval within a unified architecture. OmniBridge adopts a language-centric design that reuses pretrained LLMs and introduces a lightweight bidirectional latent alignment module. To address the challenge of task interference, we propose a two-stage decoupled training strategy: supervised fine-tuning and latent space alignment for aligning LLM behavior with multimodal reasoning, and semantic-guided diffusion training to align cross-modal latent spaces via learnable query embeddings. Extensive experiments across a wide range of benchmarks demonstrate that OmniBridge achieves competitive or state-of-the-art performance in all three tasks. Moreover, our results highlight the effectiveness of latent space alignment for unifying multimodal modeling under a shared representation space. Code and models are released at https://github.com/xiao-xt/OmniBridge.Keywords
All Related Versions
This publication has 17 references indexed in Scilit:
- JanusFlow: Harmonizing Autoregression and Rectified Flow for Unified Multimodal Understanding and Generation2025
- Janus: Decoupling Visual Encoding for Unified Multimodal Understanding and Generation2025
- R-Bench: Are Your Large Multimodal Model Robust to Real-World Corruptions?IEEE Journal of Selected Topics in Signal Processing, 2025
- MegaPairs: Massive Data Synthesis for Universal Multimodal Retrieval2025
- MMMU-Pro: A More Robust Multi-discipline Multimodal Understanding Benchmark2025
- Improve Vision Language Model Chain-of-thought Reasoning2025
- How far are we to GPT-4V? Closing the gap to commercial multimodal models with open-source suitesScience China Information Sciences, 2024
- OCRBench: on the hidden mystery of OCR in large multimodal modelsScience China Information Sciences, 2024
- Multimodal Foundation Models: From Specialists to General-Purpose AssistantsFoundations and Trends in Computer Graphics and Vision, 2024
- Flickr30k Entities: Collecting Region-to-Phrase Correspondences for Richer Image-to-Sentence Models2015