Semantics-Aware Spatial-Temporal Binaries for Cross-Modal Video Retrieval
- 9 February 2021
- journal article
- research article
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Transactions on Image Processing
- Vol. 30 (10577149) , 2989-3004
Abstract
With the current exponential growth of video-based social networks, video retrieval using natural language is receiving ever-increasing attention. Most existing approaches tackle this task by extracting individual frame-level spatial features to represent the whole video, while ignoring visual pattern consistencies and intrinsic temporal relationships across different frames. Furthermore, the semantic correspondence between natural language queries and person-centric actions in videos has not been fully explored. To address these problems, we propose a novel binary representation learning framework, named Semantics-aware Spatial-temporal Binaries ($\text{S}^{2}$ Bin), which simultaneously considers spatial-temporal context and semantic relationships for cross-modal video retrieval. By exploiting the semantic relationships between two modalities, $\text{S}^{2}$ Bin can efficiently and effectively generate binary codes for both videos and texts. In addition, we adopt an iterative optimization scheme to learn deep encoding functions with attribute-guided stochastic training. We evaluate our model on three video datasets and the experimental results demonstrate that $\text{S}^{2}$ Bin outperforms the state-of-the-art methods in terms of various cross-modal video retrieval tasks.
Keywords
Funding Information
- Foundation for Innovative Research Groups through the National Natural Science Foundation of China (61421003)
- Beihang University
This publication has 89 references indexed in Scilit:
- Latent semantic sparse hashing for cross-modal similarity search2014
- Multimodal Similarity-Preserving HashingIEEE Transactions on Pattern Analysis and Machine Intelligence, 2014
- Convolutional Neural Networks for Sentence Classification2014
- Submodular video hashing2012
- Multiple feature hashing for real-time large scale near-duplicate video retrieval2011
- NLTK2002
- A fuzzy video content representation for video summarization and content-based retrievalSignal Processing, 2000
- Techniques and systems for image and video retrievalIEEE Transactions on Knowledge and Data Engineering, 1999
- Long Short-Term MemoryNeural Computation, 1997
- An integrated system for content-based video retrieval and browsingPattern Recognition, 1997