Saliency Prediction on Omnidirectional Images with Generative Adversarial Imitation Learning
Authors:
- Xiaoming TaoXTXiaoming Tao
- Yiping DuanYDYiping Duan
- Zulin WangZWZulin Wang
Preprint
Abstract
When watching omnidirectional images (ODIs), subjects can access different viewports by moving their heads. Therefore, it is necessary to predict subjects' head fixations on ODIs. Inspired by generative adversarial imitation learning (GAIL), this paper proposes a novel approach to predict saliency of head fixations on ODIs, named SalGAIL. First, we establish a dataset for attention on ODIs (AOI). In contrast to traditional datasets, our AOI dataset is large-scale, which contains the head fixations of 30 subjects viewing 600 ODIs. Next, we mine our AOI dataset and determine three findings: (1) The consistency of head fixations are consistent among subjects, and it grows alongside the increased subject number; (2) The head fixations exist with a front center bias (FCB); and (3) The magnitude of head movement is similar across subjects. According to these findings, our SalGAIL approach applies deep reinforcement learning (DRL) to predict the head fixations of one subject, in which GAIL learns the reward of DRL, rather than the traditional human-designed reward. Then, multi-stream DRL is developed to yield the head fixations of different subjects, and the saliency map of an ODI is generated via convoluting predicted head fixations. Finally, experiments validate the effectiveness of our approach in predicting saliency maps of ODIs, significantly better than 10 state-of-the-art approaches.Keywords
All Related Versions
This publication has 19 references indexed in Scilit:
- The prediction of head and eye movement for 360 degree imagesSignal Processing: Image Communication, 2018
- SalNet360: Saliency maps for omni-directional images with CNNSignal Processing: Image Communication, 2018
- 360-aware saliency estimation with conventional image saliency predictorsSignal Processing: Image Communication, 2018
- A saliency prediction model on 360 degree images using color dictionary based sparse representationSignal Processing: Image Communication, 2018
- GBVS360, BMS360, ProSal: Extending existing saliency prediction models from 2D to omnidirectional imagesSignal Processing: Image Communication, 2018
- A feature-based approach for saliency estimation of omni-directional imagesSignal Processing: Image Communication, 2018
- Video processing for panoramic streaming using HEVC and its scalable extensionsMultimedia Tools and Applications, 2016
- SUN: Top-down saliency using natural statisticsVisual Cognition, 2009
- Finite-time Analysis of the Multiarmed Bandit ProblemMachine Learning, 2002
- A model of saliency-based visual attention for rapid scene analysisIEEE Transactions on Pattern Analysis and Machine Intelligence, 1998