Multi-task deep visual-semantic embedding for video thumbnail selection
Authors:
- 1 June 2015
- conference paper
- Published by Institute of Electrical and Electronics Engineers (IEEE)
- No. 10636919,p. 3707-3715
Abstract
Given the tremendous growth of online videos, video thumbnail, as the common visualization form of video content, is becoming increasingly important to influence user's browsing and searching experience. However, conventional methods for video thumbnail selection often fail to produce satisfying results as they ignore the side semantic information (e.g., title, description, and query) associated with the video. As a result, the selected thumbnail cannot always represent video semantics and the click-through rate is adversely affected even when the retrieved videos are relevant. In this paper, we have developed a multi-task deep visual-semantic embedding model, which can automatically select query-dependent video thumbnails according to both visual and side information. Different from most existing methods, the proposed approach employs the deep visual-semantic embedding model to directly compute the similarity between the query and video thumbnails by mapping them into a common latent semantic space, where even unseen query-thumbnail pairs can be correctly matched. In particular, we train the embedding model by exploring the large-scale and freely accessible click-through video and image data, as well as employing a multi-task learning strategy to holistically exploit the query-thumbnail relevance from these two highly related datasets. Finally, a thumbnail is selected by fusing both the representative and query relevance scores. The evaluations on 1,000 query-thumbnail dataset labeled by 191 workers in Amazon Mechanical Turk have demonstrated the effectiveness of our proposed method.Keywords
This publication has 19 references indexed in Scilit:
- Instant Mobile Video Search With Layered Audio-Video Indexing and Progressive TransmissionIEEE Transactions on Multimedia, 2014
- Efficient multi-task feature learning with calibration2014
- Click-through-based cross-view learning for image search2014
- Glove: Global Vectors for Word Representation2014
- Multimedia search rerankingACM Computing Surveys, 2014
- A Multi-View Embedding Space for Modeling Internet Images, Tags, and Their SemanticsInternational Journal of Computer Vision, 2013
- Story-Driven Summarization for Egocentric Video2013
- Query sensitive dynamic web video thumbnail generation2011
- Towards Extracting Semantically Meaningful Key Frames From Personal Video Clips: From Humans to ComputersIEEE Transactions on Circuits and Systems for Video Technology, 2008
- Making a Long Video Short: Dynamic Video Synopsis2006