Supersaliency: A Novel Pipeline for Predicting Smooth Pursuit-Based Attention Improves Generalisability of Video Saliency
Open Access
- 23 December 2019
- journal article
- research article
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Access
- Vol. 8, 1276-1289
Abstract
Predicting attention is a popular topic at the intersection of human and computer vision. However, even though most of the available video saliency data sets and models claim to target human observers’ fixations, they fail to differentiate them from smooth pursuits (SPs), a major eye movement type that is unique to perception of dynamic scenes. In this work, we strive for a more meaningful prediction and conceptual understanding of saliency in general. Because of the higher attentional selectivity of smooth pursuit compared to fixations modelled in traditional saliency research, we refer to the problem of SP prediction as ’supersaliency“. To make this distinction explicit, we (i) use algorithmic and manual annotations of SPs and fixations for two well-established video saliency data sets, (ii) train Slicing Convolutional Neural Networks for saliency prediction on either fixation-or SP-salient locations, and (iii) evaluate our and 26 publicly available dynamic saliency models on three data sets against traditional saliency and supersaliency ground truth. Overall, our models outperform the state of the art in both the new supersaliency and the traditional saliency problem settings, for which literature models are optimised. Importantly, on two independent data sets, our supersaliency model shows greater generalisation ability than its counterpart saliency model and outperforms all other models, even for fixation prediction. Furthermore, we tested an end-to-end video saliency model, which also showed systematic improvements when smooth pursuit was predicted either exclusively or together with fixations, with the best performance achieved when the model was trained for the supersaliency problem. This demonstrates the practical benefits and the potential of principled training data selection based on eye movement analysis.Keywords
All Related Versions
Funding Information
- Elitenetzwerk Bayern
This publication has 71 references indexed in Scilit:
- Recognizing the Degree of Human Attention Using EEG Signals from Mobile SensorsSensors, 2013
- Pupil dilation deconvolution reveals the dynamics of attention at high temporal resolutionProceedings of the National Academy of Sciences, 2012
- Saliency from hierarchical adaptation through decorrelation and variance normalizationImage and Vision Computing, 2012
- Keep your eyes on the ball: smooth pursuit eye movements enhance prediction of visual motionJournal of Neurophysiology, 2011
- Mind‐wandering While Reading: Attentional Decoupling, Mindless Reading and the Cascade Model of InattentionLanguage and Linguistics Compass, 2011
- Static and space-time visual saliency detection by self-resemblanceJournal of Vision, 2009
- Quantifying center bias of observers in free viewing of dynamic natural scenesJournal of Vision, 2009
- Going AWOL in the Brain: Mind Wandering Reduces Cortical Analysis of External EventsJournal of Cognitive Neuroscience, 2008
- The central fixation bias in scene viewing: Selecting an optimal viewing position independently of motor biases and image feature distributionsJournal of Vision, 2007
- Eye movements and the control of actions in everyday lifeProgress in Retinal and Eye Research, 2006