Cross-Modal and Hierarchical Modeling of Video and Text
Preprint
Abstract
Visual data and text data are composed of information at multiple granularities. A video can describe a complex scene that is composed of multiple clips or shots, where each depicts a semantically coherent event or action. Similarly, a paragraph may contain sentences with different topics, which collectively conveys a coherent message or story. In this paper, we investigate the modeling techniques for such hierarchical sequential data where there are correspondences across multiple modalities. Specifically, we introduce hierarchical sequence embedding (HSE), a generic model for embedding sequential data of different modalities into hierarchically semantic spaces, with either explicit or implicit correspondence information. We perform empirical studies on large-scale video and paragraph retrieval datasets and demonstrated superior performance by the proposed methods. Furthermore, we examine the effectiveness of our learned embeddings when applied to downstream tasks. We show its utility in zero-shot action recognition and video captioning.Keywords
All Related Versions
This publication has 31 references indexed in Scilit:
- Temporal Segment Networks: Towards Good Practices for Deep Action Recognition2016
- Video Summarization with Long Short-Term Memory2016
- Convolutional Two-Stream Network Fusion for Video Action Recognition2016
- Hierarchical Recurrent Neural Encoder for Video Representation with Application to Captioning2016
- FaceNet: A unified embedding for face recognition and clustering2015
- ActivityNet: A large-scale video benchmark for human activity understanding2015
- Deep visual-semantic alignments for generating image descriptions2015
- Show and tell: A neural image caption generator2015
- Large-Scale Video Classification with Convolutional Neural Networks2014
- Microsoft COCO: Common Objects in Context2014