Transform-Equivariant Consistency Learning for Temporal Sentence Grounding
Open Access
- 11 January 2024
- journal article
- research article
- Published by Association for Computing Machinery (ACM) in ACM Transactions on Multimedia Computing, Communications, and Applications
- Vol. 20 (4) , 1-19
Abstract
This paper addresses the temporal sentence grounding (TSG). Although existing methods have made decent achievements in this task, they not only severely rely on abundant video-query paired data for training, but also easily fail into the dataset distribution bias. To alleviate these limitations, we introduce a novel Equivariant Consistency Regulation Learning (ECRL) framework to learn more discriminative query-related frame-wise representations for each video, in a self-supervised manner. Our motivation comes from that the temporal boundary of the query-guided activity should be consistently predicted under various video-level transformations. Concretely, we first design a series of spatio-temporal augmentations on both foreground and background video segments to generate a set of synthetic video samples. In particular, we devise a self-refine module to enhance the completeness and smoothness of the augmented video. Then, we present a novel self-supervised consistency loss (SSCL) applied on the original and augmented videos to capture their invariant query-related semantic by minimizing the KL-divergence between the sequence similarity of two videos and a prior Gaussian distribution of timestamp distance. At last, a shared grounding head is introduced to predict the transform-equivariant query-guided segment boundaries for both the original and augmented videos. Extensive experiments on three challenging datasets (ActivityNet, TACoS, and Charades-STA) demonstrate both effectiveness and efficiency of our proposed ECRL framework.Keywords
All Related Versions
This publication has 57 references indexed in Scilit:
- Quo Vadis, Action Recognition? A New Model and the Kinetics Dataset2017
- Early Embedding and Late Reranking for Video Captioning2016
- Hollywood in Homes: Crowdsourcing Data Collection for Activity Understanding2016
- Shuffle and Learn: Unsupervised Learning Using Temporal Order Verification2016
- Colorful Image Colorization2016
- Unsupervised Visual Representation Learning by Context Prediction2015
- Learning Spatiotemporal Features with 3D Convolutional Networks2015
- Video co-summarization: Video summarization by visual co-occurrence2015
- Memory recall based video searchACM Transactions on Multimedia Computing, Communications, and Applications, 2014
- Bidirectional recurrent neural networksIEEE Transactions on Signal Processing, 1997