Multimodal Video-to-Near-Scene Annotation
Authors:
- 28 September 2016
- journal article
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Transactions on Multimedia
- Vol. 19 (2) , 354-366
Abstract
Traditional video annotation approaches focus on annotating keyframes/shots or whole videos with semantic keywords. However, the extraction processes of keyframes/shots might lack semantic meanings, and it is hard to use a few keywords to describe the content of a long video with multiple topics. In this work, near-scenes, which contain similar concepts, topics, or semantic meanings, are designed for better video content understanding and annotation. We propose a novel framework of hierarchical video-to-near-scene annotation not only to preserve but also to purify the semantic meanings of near-scenes. To detect near-scenes, a pattern-based prefix tree is first constructed to fast retrieve near-duplicate videos. Then, the videos containing similar near-duplicate segments and similar keywords are clustered with consideration of multimodal features including visual and textual features. To enhance the precision of near-scene detection, a pattern-to-intensity-mark (PIM) method is proposed to perform precise frame-level near-duplicate segment alignment. For each near-scene, a video-to-concept distribution model is designed to analyze the representativeness of keywords and discriminations of clusters by the proposed potential term frequency and inverse document frequency and entropy. Tags are ranked according to video-to-concept distribution scores, and the tags with the highest scores are propagated to near-scenes detected. Extensive experiments demonstrate that the proposed PIM outperforms state-of-the-art approaches compared in terms of quality segments and quality frames for near-scene detection. Furthermore, the proposed framework of hierarchical video-to-near-scene annotation can achieve high quality of near-scene annotation in terms of mean average precision.Keywords
Funding Information
- ICTL (ICTL-103-Q528)
- ATU (ATU-103-W958)
- Ministry of Science and Technology-Taiwan (MOST-103-2218-E-009-020, MOST-103-2221-E-009-154, MOST-105-2221-E-009-065)
This publication has 52 references indexed in Scilit:
- A discriminative CNN video representation for event detection2015
- An interactive tool for manual, semi-automatic and automatic video annotationComputer Vision and Image Understanding, 2015
- Active Frame, Location, and Detector Selection for Automated and Manual Video Annotation2014
- Effective Multiple Feature Hashing for Large-Scale Near-Duplicate Video RetrievalIEEE Transactions on Multimedia, 2013
- Video Copy-Detection and Localization with a Scalable Cascading FrameworkIEEE MultiMedia, 2012
- Structure Tensor Series-Based Large Scale Near-Duplicate Video RetrievalIEEE Transactions on Multimedia, 2012
- A Robust and Fast Video Copy Detection System Using Content-Based FingerprintingIEEE Transactions on Information Forensics and Security, 2010
- Automatic Video Annotation and Retrieval Based on Bayesian InferenceLecture Notes in Computer Science, 2006
- An Automated Video Annotation System2005
- Statistical models for automatic video annotation and retrieval2004