e5-omni: Explicit Cross-modal Alignment for Omni-modal Embeddings
Preprint
Abstract
Modern information systems often involve different types of items, e.g., a text query, an image, a video clip, or an audio segment. This motivates omni-modal embedding models that map heterogeneous modalities into a shared space for direct comparison. However, most recent omni-modal embeddings still rely heavily on implicit alignment inherited from pretrained vision-language model (VLM) backbones. In practice, this causes three common issues: (i) similarity logits have modality-dependent sharpness, so scores are not on a consistent scale; (ii) in-batch negatives become less effective over time because mixed-modality batches create an imbalanced hardness distribution; as a result, many negatives quickly become trivial and contribute little gradient; and (iii) embeddings across modalities show mismatched first- and second-order statistics, which makes rankings less stable. To tackle these problems, we propose e5-omni, a lightweight explicit alignment recipe that adapts off-the-shelf VLMs into robust omni-modal embedding models. e5-omni combines three simple components: (1) modality-aware temperature calibration to align similarity scales, (2) a controllable negative curriculum with debiasing to focus on confusing negatives while reducing the impact of false negatives, and (3) batch whitening with covariance regularization to better match cross-modal geometry in the shared embedding space. Experiments on MMEB-V2 and AudioCaps show consistent gains over strong bi-modal and omni-modal baselines, and the same recipe also transfers well to other VLM backbones. We release our model checkpoint at https://huggingface.co/Haon-Chen/e5-omni-7B.Keywords
All Related Versions
This publication has 9 references indexed in Scilit:
- UniME-V2: MLLM-as-a-Judge for Universal Multimodal Embedding LearningProceedings of the AAAI Conference on Artificial Intelligence, 2026
- CAFE: Unifying Representation and Generation with Contrastive-Autoregressive Finetuning2025
- Tevatron 2.0: Unified Document Retrieval Toolkit across Scale, Language, and Modality2025
- Molmo and PixMo: Open Weights and Open Data for State-of-the-Art Vision-Language Models2025
- mmE5: Improving Multimodal Multilingual Embeddings via High-quality Synthetic Data2025
- M3-Embedding: Multi-Linguality, Multi-Functionality, Multi-Granularity Text Embeddings Through Self-Knowledge Distillation2024
- AudioCaps: Generating Captions for Audios in The Wild2019
- Deep CORAL: Correlation Alignment for Deep Domain Adaptation2016
- MSR-VTT: A Large Video Description Dataset for Bridging Video and Language2016