BinauralFlow: A Causal and Streamable Approach for High-Quality Binaural Speech Synthesis with Flow Matching Models
Preprint
Abstract
Binaural rendering aims to synthesize binaural audio that mimics natural hearing based on a mono audio and the locations of the speaker and listener. Although many methods have been proposed to solve this problem, they struggle with rendering quality and streamable inference. Synthesizing high-quality binaural audio that is indistinguishable from real-world recordings requires precise modeling of binaural cues, room reverb, and ambient sounds. Additionally, real-world applications demand streaming inference. To address these challenges, we propose a flow matching based streaming binaural speech synthesis framework called BinauralFlow. We consider binaural rendering to be a generation problem rather than a regression problem and design a conditional flow matching model to render high-quality audio. Moreover, we design a causal U-Net architecture that estimates the current audio frame solely based on past information to tailor generative models for streaming inference. Finally, we introduce a continuous inference pipeline incorporating streaming STFT/ISTFT operations, a buffer bank, a midpoint solver, and an early skip schedule to improve rendering continuity and speed. Quantitative and qualitative evaluations demonstrate the superiority of our method over SOTA approaches. A perceptual study further reveals that our model is nearly indistinguishable from real-world recordings, with a $42\%$ confusion rate.
Keywords
All Related Versions
This publication has 21 references indexed in Scilit:
- Speech Enhancement and Dereverberation With Diffusion-Based Generative ModelsIEEE/ACM Transactions on Audio, Speech, and Language Processing, 2023
- Augmented/Mixed Reality Audio for Hearables: Sensing, control, and renderingIEEE Signal Processing Magazine, 2022
- SoundSpaces: Audio-Visual Navigation in 3D Environments2020
- NeRF: Representing Scenes as Neural Radiance Fields for View Synthesis2020
- The effects of spatial auditory and visual cues on mixed reality remote collaborationJournal on Multimodal User Interfaces, 2020
- Building and evaluation of a real room impulse response datasetIEEE Journal of Selected Topics in Signal Processing, 2019
- Room Impulse Response Interpolation Using a Sparse Spatio-Temporal Representation of the Sound FieldIEEE/ACM Transactions on Audio, Speech, and Language Processing, 2017
- Special issue on spatial sound in virtual, augmented, and mixed-reality environmentsVirtual Reality, 2015
- Natural Sound Rendering for Headphones: Integration of signal processing techniquesIEEE Signal Processing Magazine, 2015
- Fifty Years of Artificial ReverberationIEEE Transactions on Audio, Speech, and Language Processing, 2012