Bootstrapping Audio-Visual Video Segmentation by Strengthening Audio Cues
- 25 October 2024
- journal article
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Transactions on Circuits and Systems for Video Technology
- Vol. 35 (3) , 2398-2409
Abstract
How to effectively interact audio with vision has garnered considerable interest within the multi-modality research field. Recently, a novel audio-visual video segmentation (AVS) task has been proposed, aiming to segment the sounding objects in video frames under the guidance of audio cues. However, most existing AVS methods are hindered by a modality imbalance where the visual features tend to dominate those of the audio modality, due to a unidirectional and insufficient integration of audio cues. This imbalance skews the feature representation towards the visual aspect, impeding the learning of joint audio-visual representations and potentially causing segmentation inaccuracies. To address this issue, we propose AVSAC. Our approach features a Bidirectional Audio-Visual Decoder (BAVD) with integrated bidirectional bridges, enhancing audio cues and fostering continuous interplay between audio and visual modalities. This bidirectional interaction narrows the modality imbalance, facilitating more effective learning of integrated audio-visual representations. Additionally, we present a strategy for audio-visual frame-wise synchrony as fine-grained guidance of BAVD. This strategy enhances the share of auditory components in visual features, contributing to a more balanced audio-visual representation learning. Extensive experiments show that our method has state-of-the-art performance on several AVS public benchmarks.Keywords
All Related Versions
This publication has 31 references indexed in Scilit:
- Audio–Visual Segmentation2022
- PVT v2: Improved baselines with pyramid vision transformerComputational Visual Media, 2022
- Balanced Multimodal Learning via On-the-fly Gradient Modulation2022
- Localizing Visual Sounds the Hard Way2021
- SSTVOS: Sparse Spatiotemporal Transformers for Video Object Segmentation2021
- Improving Multispectral Pedestrian Detection by Addressing Modality Imbalance Problems2020
- Multiple Sound Sources Localization from Coarse to Fine2020
- CNN architectures for large-scale audio classification2017
- V-Net: Fully Convolutional Neural Networks for Volumetric Medical Image Segmentation2016
- Deep Residual Learning for Image Recognition2016