Semantic Pooling for Complex Event Analysis in Untrimmed Videos
- 13 September 2016
- journal article
- research article
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Transactions on Pattern Analysis and Machine Intelligence
- Vol. 39 (8) , 1617-1632
Abstract
Pooling plays an important role in generating a discriminative video representation. In this paper, we propose a new semantic pooling approach for challenging event analysis tasks (e.g., event detection, recognition, and recounting) in long untrimmed Internet videos, especially when only a few shots/segments are relevant to the event of interest while many other shots are irrelevant or even misleading. The commonly adopted pooling strategies aggregate the shots indifferently in one way or another, resulting in a great loss of information. Instead, in this work we first define a novel notion of semantic saliency that assesses the relevance of each shot with the event of interest. We then prioritize the shots according to their saliency scores since shots that are semantically more salient are expected to contribute more to the final event analysis. Next, we propose a new isotonic regularizer that is able to exploit the constructed semantic ordering information. The resulting nearly-isotonic support vector machine classifier exhibits higher discriminative power in event analysis tasks. Computationally, we develop an efficient implementation using the proximal gradient algorithm, and we prove new and closed-form proximal steps. We conduct extensive experiments on three real-world video datasets and achieve promising improvements.Keywords
Funding Information
- NIH (R01GM087694, P30DA035778)
- Data to Decisions Cooperative Research Centre
- NSFC (U1509206)
This publication has 52 references indexed in Scilit:
- Order Preserving Sparse CodingIEEE Transactions on Pattern Analysis and Machine Intelligence, 2014
- Knowledge Adaptation with PartiallyShared Features for Event DetectionUsing Few ExemplarsIEEE Transactions on Pattern Analysis and Machine Intelligence, 2014
- Proximal alternating linearized minimization for nonconvex and nonsmooth problemsMathematical Programming, 2013
- Image Classification with the Fisher Vector: Theory and PracticeInternational Journal of Computer Vision, 2013
- Recognizing Complex Events Using Large Margin Joint Low-Level Event Model2012
- Local velocity-adapted motion events for spatio-temporal recognitionComputer Vision and Image Understanding, 2007
- Distinctive Image Features from Scale-Invariant KeypointsInternational Journal of Computer Vision, 2004
- Local Extremes, Runs, Strings and MultiresolutionThe Annals of Statistics, 2001
- Nonlinear total variation based noise removal algorithmsPhysica D: Nonlinear Phenomena, 1992
- A generalized proximal point algorithm for certain non-convex minimization problemsInternational Journal of Systems Science, 1981