Gaze-enabled egocentric video summarization via constrained submodular maximization
Authors:
- 1 June 2015
- conference paper
- Published by Institute of Electrical and Electronics Engineers (IEEE) in Conference on Computer Vision and Pattern Recognition (CVPR)
- Vol. 2015 (10636919) , 2235-2244
Abstract
With the proliferation of wearable cameras, the number of videos of users documenting their personal lives using such devices is rapidly increasing. Since such videos may span hours, there is an important need for mechanisms that represent the information content in a compact form (i.e., shorter videos which are more easily browsable/sharable). Motivated by these applications, this paper focuses on the problem of egocentric video summarization. Such videos are usually continuous with significant camera shake and other quality issues. Because of these reasons, there is growing consensus that direct application of standard video summarization tools to such data yields unsatisfactory performance. In this paper, we demonstrate that using gaze tracking information (such as fixation and saccade) significantly helps the summarization task. It allows meaningful comparison of different image frames and enables deriving personalized summaries (gaze provides a sense of the camera wearer's intent). We formulate a summarization model which captures common-sense properties of a good summary, and show that it can be solved as a submodular function maximization with partition matroid constraints, opening the door to a rich body of work from combinatorial optimization. We evaluate our approach on a new gaze-enabled egocentric video dataset (over 15 hours), which will be a valuable standalone resource.Keywords
This publication has 29 references indexed in Scilit:
- Rich Feature Hierarchies for Accurate Object Detection and Semantic Segmentation2014
- Story-Driven Summarization for Egocentric Video2013
- First-Person Activity Recognition: What Are They Doing to Me?2013
- Submodular feature selection for high-dimensional acoustic score spaces2013
- Detecting activities of daily living in first-person camera views2012
- Active Visual SegmentationIEEE Transactions on Pattern Analysis and Machine Intelligence, 2011
- Figure-ground segmentation improves handled object recognition in egocentric video2010
- Creating map-based storyboards for browsing tour videos2008
- Schematic storyboarding for video visualization and editing2006
- A generic framework of user attention model and its application in video summarizationIEEE Transactions on Multimedia, 2005