DiffSal: Joint Audio and Video Learning for Diffusion Saliency Prediction
Authors:
- 16 June 2024
- conference paper
- Published by Institute of Electrical and Electronics Engineers (IEEE)
- p. 27263-27273
Abstract
Audio-visual saliency prediction can draw support from diverse modality complements, but further performance enhancement is still challenged by customized architectures as well as task-specific loss functions. In recent studies, denoising diffusion models have shown more promising in unifying task frameworks owing to their inherent ability of generalization. Following this motivation, a novel Diffusion architecture for generalized audio-visual Saliency prediction (DiffSal) is proposed in this work, which formulates the prediction problem as a conditional generative task of the saliency map by utilizing input audio and video as the conditions. Based on the spatiotemporal audio-visual features, an extra network Saliency-UNet is designed to perform multimodal attention modulation for progressive refinement of the ground-truth saliency map from the noisy map. Extensive experiments demonstrate that the proposed DiffSal can achieve excellent performance across six challenging audio-visual benchmarks, with an average relative improvement of 6.3% over the previous state-of-the-art results by six metrics. The project url is htt ps: //junwenxiong. github.io/DiffSal.Keywords
All Related Versions
This publication has 44 references indexed in Scilit:
- Unleashing Transformers: Parallel Token Prediction with Discrete Absorbing Diffusion for Fast High-Resolution Image Generation from Vector-Quantized Codes2022
- ECANet: Explicit cyclic attention-based network for video saliency predictionNeurocomputing, 2022
- Unified Image and Video Saliency Modeling2020
- Rethinking Spatiotemporal Feature Learning: Speed-Accuracy Trade-offs in Video Classification2018
- Spatiotemporal visual saliency guided perceptual high efficiency video coding with neural networkNeurocomputing, 2018
- Multimodal Saliency Models for Videos2016
- A perceptually based spatio-temporal computational framework for visual saliency estimationSignal Processing: Image Communication, 2015
- Creating Summaries from User Videos2014
- Clustering of Gaze During Dynamic Scene Viewing is Predicted by MotionCognitive Computation, 2010
- Actions in context2009