TS-Binaural: Visually Guided Binaural Audio Generation by Temporal–Spatial Dynamic Analysis
- 31 March 2026
- journal article
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Transactions on Neural Networks and Learning Systems
- Vol. PP (2162237X) , 1-15
Abstract
Visually guided binaural audio generation (VGBAG) aims at estimating spatial information from visual images to reconstruct binaural audio, yet struggles to model dynamic audio-visual relationships in complex multisource scenarios. This work focuses on musical instrument performance videos, a critical case characterized by a variable number of sound sources, multiple performance forms, and diverse stage effects. These dynamic characteristics are ignored by existing methods, resulting in inaccuracy of the generated binaural audio. To overcome these limitations, in this article, we propose a novel VGBAG method (TS-Binaural) based on temporal–spatial dynamic analysis (TSDA). Different from the overall audio-visual content analysis in previous work, the TS-Binaural employs a novel “separation-later-mixing” strategy. It splits complex audio-visual scenes into separate units to improve accuracy. The TS-Binaural comprises two key components: a TSDA module and a temporal–spatial information fusion (TSIF) guided binaural audio generation module. Furthermore, in the TSDA module, we design an automatic mask generation-based performance analyzer (AMGPA) to prevent intersource interference. The AMGPA integrates novel position detection logic to identify and separate the activation region, which refers to a separate visual region consisting of a player and his/her musical instrument being played at different times in the video. Extensive quantitative and qualitative comparisons demonstrate that our TS-Binaural outperforms the state-of-the-art VGBAG methods.Keywords
Funding Information
- National Natural Science Foundation of China (62201524)
- National Natural Science Foundation of China (CUC25CGJ01)
This publication has 44 references indexed in Scilit:
- Cross-modal generative model for visual-guided binaural stereo generationKnowledge-Based Systems, 2024
- Visually Guided Sound Source Separation Using Cascaded Opponent Filter Network2021
- Sep-Stereo: Visually Guided Stereophonic Audio Generation by Associating Source Separation2020
- 2.5D Visual Sound2019
- Dual Attention Network for Scene Segmentation2019
- The Sound of Pixels2018
- Dilated Residual Networks2017
- U-Net: Convolutional Networks for Biomedical Image Segmentation2015
- The Pascal Visual Object Classes Challenge: A RetrospectiveInternational Journal of Computer Vision, 2014
- Microsoft COCO: Common Objects in Context2014