MMMORRF: Multimodal Multilingual Modularized Reciprocal Rank Fusion
Authors:
- Saron SamuelSSSaron Samuel
- Dan DeGenaroDDDan DeGenaro
- Kate SandersKSKate Sanders
- Oluwaseun EisapeOEOluwaseun Eisape
- Eugene YangEYEugene Yang
- David EtterDEDavid Etter
Preprint
Abstract
Videos inherently contain multiple modalities, including visual events, text overlays, sounds, and speech, all of which are important for retrieval. However, state-of-the-art multimodal language models like VAST and LanguageBind are built on vision-language models (VLMs), and thus overly prioritize visual signals. Retrieval benchmarks further reinforce this bias by focusing on visual queries and neglecting other modalities. We create a search system MMMORRF that extracts text and features from both visual and audio modalities and integrates them with a novel modality-aware weighted reciprocal rank fusion. MMMORRF is both effective and efficient, demonstrating practicality in searching videos based on users' information needs instead of visual descriptive queries. We evaluate MMMORRF on MultiVENT 2.0 and TVR, two multimodal benchmarks designed for more targeted information needs, and find that it improves nDCG@20 by 81% over leading multimodal encoders and 37% over single-modality retrieval, demonstrating the value of integrating diverse modalities.Keywords
All Related Versions
This publication has 16 references indexed in Scilit:
- The Faiss LibraryIEEE Transactions on Big Data, 2025
- MultiVENT 2.0: A Massive Multilingual Benchmark for Event-Centric Video Retrieval2025
- InternVideo2: Scaling Foundation Models for Multimodal Video Understanding2024
- VALOR: Vision-Audio-Language Omni-Perception Pretraining Model and DatasetIEEE Transactions on Pattern Analysis and Machine Intelligence, 2024
- A large cross-modal video retrieval dataset with reading comprehensionPattern Recognition, 2024
- Sigmoid Loss for Language Image Pre-Training2023
- Real-Time Scene Text Detection With Differentiable Binarization and Adaptive Scale FusionIEEE Transactions on Pattern Analysis and Machine Intelligence, 2022
- Frozen in Time: A Joint Video and Image Encoder for End-to-End Retrieval2021
- MSR-VTT: A Large Video Description Dataset for Bridging Video and Language2016
- Reciprocal rank fusion outperforms condorcet and individual rank learning methods2009