Moment Alignment Transformer for Video-to-Video Moment Retrieval
Authors:
- Yogesh KumarYKYogesh Kumar
- Uday AgarwalUAUday Agarwal
- Manish GuptaMGManish Gupta
- Anand MishraAMAnand Mishra
Preprint
- 14 March 2026
- preprint
- Published by Elsevier in SSRN Electronic Journal
Abstract
Video-to-video moment retrieval (Vid2VidMR) is the task of localizing unseen events or moments in a target video using a query video. This task is challenging due to the need for semantic frame-level alignment and modeling complex dependencies between query and target videos. To address this, we introduce MATR (Moment Alignment TRansformer), a transformer-based model that captures semantic context and temporal detail for precise moment localization. MATR conditions target video representations on query features through dual-stage sequence alignment, encoding the necessary correlations and dependencies. These representations guide foreground/background classification and boundary prediction heads to identify moments in the target video that semantically match the query. To provide strong task-specific initialization, we further propose a self-supervised pre-training objective where the model learns to localize random clips within videos. Extensive experiments show that MATR achieves absolute gains of 15.4% in R@1 and 11.4% in mIoU over state-of-the-art methods on the ActivityNet-VRL dataset, and 16.4% in R@1 and 16.3% in mIoU on our newly proposed SportsMoments dataset. Code is available at: https://github.com/vl2g/MATR.Keywords
All Related Versions
This publication has 5 references indexed in Scilit:
- Multilevel spatial–temporal feature analysis for generic event boundary detection in videosComputer Vision and Image Understanding, 2025
- Semantic Relevance Learning for Video-Query Based Video Moment RetrievalIEEE Transactions on Multimedia, 2023
- Linear Sequence-to-Sequence AlignmentIEEE Transactions on Pattern Analysis and Machine Intelligence, 2008
- Dynamic programming algorithm optimization for spoken word recognitionIEEE Transactions on Acoustics, Speech, and Signal Processing, 1978
- A general method applicable to the search for similarities in the amino acid sequence of two proteinsJournal of Molecular Biology, 1970