Visually Informed Binaural Audio Generation without Binaural Audios
Authors:
- 1 June 2021
- conference paper
- Published by Institute of Electrical and Electronics Engineers (IEEE)
- p. 15480-15489
Abstract
Stereophonic audio, especially binaural audio, plays an essential role in immersive viewing environments. Recent research has explored generating visually guided stereophonic audios supervised by multi-channel audio collections. However, due to the requirement of professional recording devices, existing datasets are limited in scale and variety, which impedes the generalization of supervised methods in real-world scenarios. In this work, we propose PseudoBinaural, an effective pipeline that is free of binaural recordings. The key insight is to carefully build pseudo visual-stereo pairs with mono data for training. Specifically, we leverage spherical harmonic decomposition and head-related impulse response (HRIR) to identify the relationship between spatial locations and received binaural audios. Then in the visual modality, corresponding visual cues of the mono data are manually placed at sound source positions to form the pairs. Compared to fully-supervised paradigms, our binaural-recording-free pipeline shows great stability in cross-dataset evaluation and achieves comparable performance under subjective preference. Moreover, combined with binaural recordings, our method is able to further boost the performance of binaural audio generation under supervised settings 1 .Keywords
All Related Versions
This publication has 23 references indexed in Scilit:
- Scene-aware audio for 360° videosACM Transactions on Graphics, 2018
- Look, Listen and Learn2017
- Motion informed audio source separation2017
- Deep Residual Learning for Image Recognition2016
- Visually Indicated Sounds2016
- Fast implementation of sparse iterative covariance-based estimation for source localizationThe Journal of the Acoustical Society of America, 2012
- Speech Enhancement and Recognition in Meetings With an Audio–Visual Sensor ArrayIEEE Transactions on Audio, Speech, and Language Processing, 2007
- Geomagnetism and Schmidt quasi-normalizationGeophysical Journal International, 2005
- The CIPIC HRTF database2002
- A Note on the Concept of the Visual Field in Neurology, Psychology, and Visual NeurosciencePerception, 1996