Saliency Prediction on Omnidirectional Image With Generative Adversarial Imitation Learning
- 18 January 2021
- journal article
- research article
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Transactions on Image Processing
- Vol. 30 (10577149) , 2087-2102
Abstract
When watching omnidirectional images (ODIs), subjects can access different viewports by moving their heads. Therefore, it is necessary to predict subjects’ head fixations on ODIs. Inspired by generative adversarial imitation learning (GAIL), this paper proposes a novel approach to predict saliency of head fixations on ODIs, named SalGAIL. First, we establish a dataset for attention on ODIs (AOI). In contrast to traditional datasets, our AOI dataset is large-scale, which contains the head fixations of 30 subjects viewing 600 ODIs. Next, we mine our AOI dataset and discover three findings: (1) the consistency of head fixations are consistent among subjects, and it grows alongside the increased subject number; (2) the head fixations exist with a front center bias (FCB); and (3) the magnitude of head movement is similar across the subjects. According to these findings, our SalGAIL approach applies deep reinforcement learning (DRL) to predict the head fixations of one subject, in which GAIL learns the reward of DRL, rather than the traditional human-designed reward. Then, multi-stream DRL is developed to yield the head fixations of different subjects, and the saliency map of an ODI is generated via convoluting predicted head fixations. Finally, experiments validate the effectiveness of our approach in predicting saliency maps of ODIs, significantly better than 11 state-of-the-art approaches. Our AOI dataset and code of SalGAIL are available online at https://github.com/yanglixiaoshen/SalGAIL .Keywords
All Related Versions
Funding Information
- Beijing Natural Science Foundation (JQ20020)
- NSFC Projects (61876013, 61922009)
This publication has 46 references indexed in Scilit:
- SALICON: Reducing the Semantic Gap in Saliency Prediction by Adapting Deep Neural Networks2015
- A Framework to Evaluate Omnidirectional Video Coding Schemes2015
- Exploiting Surroundedness for Saliency Detection: A Boolean Map ApproachIEEE Transactions on Pattern Analysis and Machine Intelligence, 2015
- Saliency detection by multi-context deep learning2015
- Global Contrast Based Salient Region DetectionIEEE Transactions on Pattern Analysis and Machine Intelligence, 2014
- Boosting bottom-up and top-down visual features for saliency estimation2012
- SUN: Top-down saliency using natural statisticsVisual Cognition, 2009
- Identifying fixations and saccades in eye-tracking protocols2000
- A model of saliency-based visual attention for rapid scene analysisIEEE Transactions on Pattern Analysis and Machine Intelligence, 1998
- Q-learningMachine Learning, 1992