Disentangling speech from surroundings with neural embeddings
Preprint
Abstract
We present a method to separate speech signals from noisy environments in the embedding space of a neural audio codec. We introduce a new training procedure that allows our model to produce structured encodings of audio waveforms given by embedding vectors, where one part of the embedding vector represents the speech signal, and the rest represent the environment. We achieve this by partitioning the embeddings of different input waveforms and training the model to faithfully reconstruct audio from mixed partitions, thereby ensuring each partition encodes a separate audio attribute. As use cases, we demonstrate the separation of speech from background noise or from reverberation characteristics. Our method also allows for targeted adjustments of the audio output characteristics.Keywords
All Related Versions
This publication has 8 references indexed in Scilit:
- NVC-Net: End-To-End Adversarial Voice Conversion2022
- SoundStream: An End-to-End Neural Audio CodecIEEE/ACM Transactions on Audio, Speech, and Language Processing, 2021
- Speech Resynthesis from Discrete Disentangled Self-Supervised Representations2021
- SPICE: Self-Supervised Pitch EstimationIEEE/ACM Transactions on Audio, Speech, and Language Processing, 2020
- Multi-target Voice Conversion without Parallel Data by Adversarially Learning Disentangled Audio Representations2018
- ViSQOL: an objective speech quality modelEURASIP Journal on Audio, Speech, and Music Processing, 2015
- LibriVox: Free Public Domain AudiobooksReference Reviews, 2014
- A Sober Look at the Unsupervised Learning of Disentangled Representations and their Evaluation