Semi-supervised Video Paragraph Grounding with Contrastive Encoder
Authors:
- 1 June 2022
- conference paper
- Published by Institute of Electrical and Electronics Engineers (IEEE)
- p. 2456-2465
Abstract
Video events grounding aims at retrieving the most relevant moments from an untrimmed video in terms of a given natural language query. Most previous works focus on Video Sentence Grounding (VSG), which localizes the moment with a sentence query. Recently, researchers extended this task to Video Paragraph Grounding (VPG) by retrieving multiple events with a paragraph. However, we find the existing VPG methods may not perform well on context modeling and highly rely on video-paragraph annotations. To tackle this problem, we propose a novel VPG method termed Semi-supervised Video-Paragraph TRansformer (SVPTR), which can more effectively exploit contextual information in paragraphs and significantly reduce the dependency on annotated data. Our SVPTR method consists of two key components: (1) a base model VPTR that learns the video-paragraph alignment with contrastive encoders and tackles the lack of sentence-level contextual interactions and (2) a semi-supervised learning framework with multimodal feature perturbations that reduces the requirements of annotated training data. We evaluate our model on three widely-used video grounding datasets, i.e., ActivityNet-Caption, Charades-CD-OOD, and TACoS. The experimental results show that our SVPTR method establishes the new state-of-the-art performance on all datasets. Even under the conditions of fewer annotations, it can also achieve competitive results compared with recent VPG methods.Keywords
Funding Information
- National Natural Science Foundation of China (61976049,62072080)
This publication has 42 references indexed in Scilit:
- Attentive Moment Retrieval in Videos2018
- Bidirectional Attentive Fusion with Context Gating for Dense Video Captioning2018
- Localizing Moments in Video with Natural Language2017
- TALL: Temporal Activity Localization via Language Query2017
- Dense-Captioning Events in Videos2017
- Beam Search Strategies for Neural Machine Translation2017
- Context Encoders: Feature Learning by Inpainting2016
- Learning Spatiotemporal Features with 3D Convolutional Networks2015
- Recognizing Fine-Grained and Composite Activities Using Hand-Centric Features and Script DataInternational Journal of Computer Vision, 2015
- Glove: Global Vectors for Word Representation2014