Query - Dependent Video Representation for Moment Retrieval and Highlight Detection
Authors:
Top Cited Papers
- 1 June 2023
- conference paper
- Published by Institute of Electrical and Electronics Engineers (IEEE)
- p. 23023-23033
Abstract
Recently, video moment retrieval and highlight detection (MR/HD) are being spotlighted as the demand for video understanding is drastically increased. The key objective of MR/HD is to localize the moment and estimate clip-wise accordance level, i.e., saliency score, to the given text query. Although the recent transformer-based models brought some advances, we found that these methods do not fully exploit the information of a given query. For example, the relevance between text query and video contents is sometimes neglected when predicting the moment and its saliency. To tackle this issue, we introduce Query-Dependent DETR (QD-DETR), a detection transformer tailored for MR/HD. As we observe the insignificant role of a given query in transformer architectures, our encoding module starts with cross-attention layers to explicitly inject the context of text query into video representation. Then, to enhance the model's capability of exploiting the query information, we manipulate the video-query pairs to produce irrelevant pairs. Such negative (irrelevant) video-query pairs are trained to yield low saliency scores, which in turn, encourages the model to estimate precise accordance between query-video pairs. Lastly, we present an input-adaptive saliency predictor which adaptively defines the criterion of saliency scores for the given video-query pairs. Our extensive studies verify the importance of building the query-dependent representation for MR/HD. Specifically, QD-DETR outperforms state-of-the-art methods on QVHighlights, TVSum, and Charades-STA datasets. Codes are available at github.com/wjun0830IQD-DETR.Keywords
All Related Versions
This publication has 36 references indexed in Scilit:
- Localizing Moments in Video with Natural Language2017
- Weakly Supervised Summarization of Web Videos2017
- Focal Loss for Dense Object Detection2017
- TALL: Temporal Activity Localization via Language Query2017
- To Click or Not To Click2016
- Video2GIF: Automatic Generation of Animated GIFs from Video2016
- Deep Residual Learning for Image Recognition2016
- Highlight Detection with Pairwise Deep Ranking for First-Person Video Summarization2016
- Multi-task deep visual-semantic embedding for video thumbnail selection2015
- Large-Scale Video Summarization Using Web-Image Priors2013