Multi-Modal Domain Adaptation Across Video Scenes for Temporal Video Grounding
Authors:
Preprint
Abstract
Temporal Video Grounding (TVG) aims to localize the temporal boundary of a specific segment in an untrimmed video based on a given language query. Since datasets in this domain are often gathered from limited video scenes, models tend to overfit to scene-specific factors, which leads to suboptimal performance when encountering new scenes in real-world applications. In a new scene, the fine-grained annotations are often insufficient due to the expensive labor cost, while the coarse-grained video-query pairs are easier to obtain. Thus, to address this issue and enhance model performance on new scenes, we explore the TVG task in an unsupervised domain adaptation (UDA) setting across scenes for the first time, where the video-query pairs in the source scene (domain) are labeled with temporal boundaries, while those in the target scene are not. Under the UDA setting, we introduce a novel Adversarial Multi-modal Domain Adaptation (AMDA) method to adaptively adjust the model's scene-related knowledge by incorporating insights from the target data. Specifically, we tackle the domain gap by utilizing domain discriminators, which help identify valuable scene-related features effective across both domains. Concurrently, we mitigate the semantic gap between different modalities by aligning video-query pairs with related semantics. Furthermore, we employ a mask-reconstruction approach to enhance the understanding of temporal semantics within a scene. Extensive experiments on Charades-STA, ActivityNet Captions, and YouCook2 demonstrate the effectiveness of our proposed method.Keywords
All Related Versions
This publication has 10 references indexed in Scilit:
- Multi-Modal Cross-Domain Alignment Network for Video Moment RetrievalIEEE Transactions on Multimedia, 2022
- Aligning Correlation Information for Domain Adaptation in Action RecognitionIEEE Transactions on Neural Networks and Learning Systems, 2022
- HiSA: Hierarchically Semantic Associating for Video Temporal GroundingIEEE Transactions on Image Processing, 2022
- Regularized Two Granularity Loss Function for Weakly Supervised Video Moment RetrievalIEEE Transactions on Multimedia, 2021
- Heterogeneous Graph Attention Network for Unsupervised Multiple-Target Domain AdaptationIEEE Transactions on Pattern Analysis and Machine Intelligence, 2020
- Unsupervised Domain Adaptation via Discriminative Manifold PropagationIEEE Transactions on Pattern Analysis and Machine Intelligence, 2020
- Span-based Localizing Network for Natural Language Video Localization2020
- Generalized Conditional Domain Adaptation: A Causal Perspective With Low-Rank TranslatorsIEEE Transactions on Cybernetics, 2018
- Grounding Action Descriptions in VideosTransactions of the Association for Computational Linguistics, 2013
- Domain Adaptation via Transfer Component AnalysisIEEE Transactions on Neural Networks, 2010