Instance-Level Panoramic Audio-Visual Saliency Detection and Ranking
Open Access
- 28 October 2024
- conference paper
- Published by Association for Computing Machinery (ACM)
- p. 9426-9434
Abstract
Panoramic audio-visual saliency detection is to segment the most attention-attractive regions in 360° panoramic videos with sound. To meticulously delineate the detected salient regions and effectively model human attention shift, we extend this task to more fine-grained instance scenarios: identifying salient object instances and inferring their saliency ranks. In this paper, we propose the first instance-level framework that can simultaneously be applied to segmentation and ranking of multiple salient objects in panoramic videos. Specifically, it consists of a distortion-aware pixel decoder to overcome panoramic distortions, a sequential audio-visual fusion module to integrate audio-visual information, and a spatio-temporal object decoder to separate individual instances and predict their saliency scores. Moreover, owing to the absence of such annotations, we create the ground-truth saliency ranks for the PAVS10K benchmark. Extensive experiments demonstrate that our model is capable of achieving state-of-the-art performance on the PAVS10K for both saliency detection and ranking tasks. The code is available at https://github.com/ruohaoguo/pavsodr.Keywords
Funding Information
- National Natural Science Foundation of China (62371009 &61971008)
This publication has 34 references indexed in Scilit:
- Audio–Visual Segmentation2022
- Highly Efficient Salient Object Detection with 100K Parameters2020
- Suppress and Balance: A Simple Gated Network for Salient Object Detection2020
- See More, Know More: Unsupervised Video Object Segmentation With Co-Attention Siamese Networks2019
- Cascaded Partial Decoder for Fast and Accurate Salient Object Detection2019
- Pyramid Dilated Deeper ConvLSTM for Video Salient Object Detection2018
- Distortion-Aware Convolutional Filters for Dense Prediction in Panoramic Images2018
- Instance-Level Salient Object Segmentation2017
- Deep Residual Learning for Image Recognition2016
- Submodular Salient Region Detection2013