Audio-visual Saliency for Omnidirectional Videos
Preprint
Abstract
Visual saliency prediction for omnidirectional videos (ODVs) has shown great significance and necessity for omnidirectional videos to help ODV coding, ODV transmission, ODV rendering, etc.. However, most studies only consider visual information for ODV saliency prediction while audio is rarely considered despite its significant influence on the viewing behavior of ODV. This is mainly due to the lack of large-scale audio-visual ODV datasets and corresponding analysis. Thus, in this paper, we first establish the largest audio-visual saliency dataset for omnidirectional videos (AVS-ODV), which comprises the omnidirectional videos, audios, and corresponding captured eye-tracking data for three video sound modalities including mute, mono, and ambisonics. Then we analyze the visual attention behavior of the observers under various omnidirectional audio modalities and visual scenes based on the AVS-ODV dataset. Furthermore, we compare the performance of several state-of-the-art saliency prediction models on the AVS-ODV dataset and construct a new benchmark. Our AVS-ODV datasets and the benchmark will be released to facilitate future research.Keywords
All Related Versions
This publication has 12 references indexed in Scilit:
- AIGCIQA2023: A Large-Scale Image Quality Assessment Database for AI Generated Images: From the Perspectives of Quality, Authenticity and Correspondence2024
- Perceptual Quality Assessment of Omnidirectional Audio-Visual Signals2024
- Attentive Deep Image Quality Assessment for Omnidirectional StitchingIEEE Journal of Selected Topics in Signal Processing, 2023
- Confusing Image Quality Assessment: Toward Better Augmented Reality ExperienceIEEE Transactions on Image Processing, 2022
- The prediction of head and eye movement for 360 degree imagesSignal Processing: Image Communication, 2018
- Predicting Head Movement in Panoramic Video: A Deep Reinforcement Learning ApproachIEEE Transactions on Pattern Analysis and Machine Intelligence, 2018
- Predicting Human Eye Fixations via an LSTM-Based Saliency Attentive ModelIEEE Transactions on Image Processing, 2018
- Global Contrast Based Salient Region DetectionIEEE Transactions on Pattern Analysis and Machine Intelligence, 2014
- Video Saliency Incorporating Spatiotemporal Cues and Uncertainty WeightingIEEE Transactions on Image Processing, 2014
- A model of saliency-based visual attention for rapid scene analysisIEEE Transactions on Pattern Analysis and Machine Intelligence, 1998