Axel Roebel
Current Organizations / Institutes
The organizations/institutes from the scholar's latest article
Previous Organizations / Institutes
The organizations / institutes from the scholar's other articles in the past ten years
Research Subjects
Axel Roebel has a total of 33 co-authors. Here are the top co-authors by number of shared publications:
- Frederik Bous
Sorbonne Universit, CNRS,UMR 9912 STMS, IRCAM,Paris,France
4 shared publications - Nicolas Obin
STMS Lab, IRCAM, CNRS, Sorbonne Université
3 shared publications - Gilles Degottex
IRCAM - CNRS-UMR9912-STMS, Analysis-Synthesis Team, 1, place Igor Stravinsky, 75004 Paris, France
2 shared publications - Guillaume Doras
IRCAM 1, place Igor-Stravinsky 75004 Paris France
2 shared publications - Laurent Benaroya
Analysis/Synthesis Team—STMS, IRCAM, Sorbonne University, CNRS, French Ministry of Culture, 75004 Paris, France
2 shared publications Luc Ardaillon
2 shared publications
Latest Publications
- Open Access
Controlled oscillation modeling using port-Hamiltonian neural networks
Learning dynamical systems through purely data-driven methods is challenging as they do not learn the underlying conservation laws that enable them to correctly generalize. Existing port-Hamiltonian neural network methods have recently been successfully applied for modeling mechanical systems. However, even though these methods are designed on power-balance principles, they usually do not consider power-preserving discretizations and often rely on Runge-Kutta numerical methods. In this work, we propose to use a second-order discrete gradient method embedded in the learning of dynamical systems with port-Hamiltonian neural networks. Numerical results are provided for three systems deliberately selected to span different ranges of dynamical behavior under control: a baseline harmonic oscillator with quadratic energy storage; a Duffing oscillator, with a non-quadratic Hamiltonian offering amplitude-dependent effects; and a self-sustained oscillator, which can stabilize in a controlled limit cycle through the incorporation of a nonlinear dissipation. We show how the use of this discrete gradient method outperforms the performance of a Runge-Kutta method of the same order. Experiments are also carried out to compare two theoretically equivalent port-Hamiltonian systems formulations and to analyze the impact of regularizing the Jacobian of port-Hamiltonian neural networks during training.
Open AccessControlled oscillation modeling using port-Hamiltonian neural networks
Learning dynamical systems through purely data-driven methods is challenging as they do not learn the underlying conservation laws that enable them to correctly generalize. Existing port-Hamiltonian neural network methods have recently been successfully applied for modeling mechanical systems. However, even though these methods are designed on power-balance principles, they usually do not consider power-preserving discretizations and often rely on Runge-Kutta numerical methods. In this work, we propose to use a second-order discrete gradient method embedded in the learning of dynamical systems with port-Hamiltonian neural networks. Numerical results are provided for three systems deliberately selected to span different ranges of dynamical behavior under control: a baseline harmonic oscillator with quadratic energy storage; a Duffing oscillator, with a non-quadratic Hamiltonian offering amplitude-dependent effects; and a self-sustained oscillator, which can stabilize in a controlled limit cycle through the incorporation of a nonlinear dissipation. We show how the use of this discrete gradient method outperforms the performance of a Runge-Kutta method of the same order. Experiments are also carried out to compare two theoretically equivalent port-Hamiltonian systems formulations and to analyze the impact of regularizing the Jacobian of port-Hamiltonian neural networks during training.
- Open Access
- 12 September 2023
- Published by Audio Engineering Society in Journal of the Audio Engineering Society
- Vol. 71 (9) , 552-565
DDSP-Piano: A Neural Sound Synthesizer Informed by Instrument Knowledge
Instrument sound synthesis using deep neural networks has received numerous improvements over the last couple of years. Among them, the Differentiable Digital Signal Processing (DDSP) framework has modernized the spectral modeling paradigm by including signal-based synthesizers and effects into fully differentiable architectures. The present work extends the applications of DDSP to the task of polyphonic sound synthesis, with the proposal of a differentiable piano synthesizer conditioned on...
- 12 September 2023
- Published by Audio Engineering Society in Journal of the Audio Engineering Society
- Vol. 71 (9) , 552-565
Open AccessDDSP-Piano: A Neural Sound Synthesizer Informed by Instrument Knowledge
Instrument sound synthesis using deep neural networks has received numerous improvements over the last couple of years. Among them, the Differentiable Digital Signal Processing (DDSP) framework has modernized the spectral modeling paradigm by including signal-based synthesizers and effects into fully differentiable architectures. The present work extends the applications of DDSP to the task of polyphonic sound synthesis, with the proposal of a differentiable piano synthesizer conditioned on...
- Open Access
- 4 June 2023
- Published by Institute of Electrical and Electronics Engineers (IEEE)
- Vol. 50, 1-5
Analysis and Transformation of Voice Level in Singing Voice
We introduce a neural auto-encoder that transforms the musical dynamic in recordings of singing voice via changes in voice level. Since most recordings of singing voice are not annotated with voice level we propose a means to estimate the voice level from the signal’s timbre using a neural voice level estimator. We introduce the recording factor that relates the voice level to the recorded signal power as a proportionality constant. This unknown constant depends on the recording conditions and the post-processing and may thus be different for each recording (but is constant across each recording). We provide two approaches to estimate the voice level without knowing the recording factor. The unknown recording factor can either be learned alongside the weights of the voice level estimator, or a special loss function based on the scalar product can be used to only match the contour of the recorded signal’s power. The voice level models are used to condition a previously introduced bottleneck auto-encoder that disentangles its input, the mel-spectrogram, from the voice level. We evaluate the voice level models on recordings annotated with musical dynamic and by their ability to provide useful information to the auto-encoder. A perceptive test is carried out that evaluates the perceived change in voice level in transformed recordings and the synthesis quality. The perceptive test confirms that changing the conditional input changes the perceived voice level accordingly thus suggesting that the proposed voice level models encode information about the true voice level.
- 4 June 2023
- Published by Institute of Electrical and Electronics Engineers (IEEE)
- Vol. 50, 1-5
Open AccessAnalysis and Transformation of Voice Level in Singing Voice
We introduce a neural auto-encoder that transforms the musical dynamic in recordings of singing voice via changes in voice level. Since most recordings of singing voice are not annotated with voice level we propose a means to estimate the voice level from the signal’s timbre using a neural voice level estimator. We introduce the recording factor that relates the voice level to the recorded signal power as a proportionality constant. This unknown constant depends on the recording conditions and the post-processing and may thus be different for each recording (but is constant across each recording). We provide two approaches to estimate the voice level without knowing the recording factor. The unknown recording factor can either be learned alongside the weights of the voice level estimator, or a special loss function based on the scalar product can be used to only match the contour of the recorded signal’s power. The voice level models are used to condition a previously introduced bottleneck auto-encoder that disentangles its input, the mel-spectrogram, from the voice level. We evaluate the voice level models on recordings annotated with musical dynamic and by their ability to provide useful information to the auto-encoder. A perceptive test is carried out that evaluates the perceived change in voice level in transformed recordings and the synthesis quality. The perceptive test confirms that changing the conditional input changes the perceived voice level accordingly thus suggesting that the proposed voice level models encode information about the true voice level.
- Open Access
Manipulating Voice Attributes by Adversarial Learning of Structured Disentangled Representations
Voice conversion (VC) consists of digitally altering the voice of an individual to manipulate part of its content, primarily its identity, while maintaining the rest unchanged. Research in neural VC has accomplished considerable breakthroughs with the capacity to falsify a voice identity using a small amount of data with a highly realistic rendering. This paper goes beyond voice identity manipulation and presents an original neural architecture that allows the manipulation of voice attributes (e.g., gender and age). The proposed architecture is inspired by the fader network, transferring the same ideas to voice manipulation. The information conveyed by the speech signal is disentangled into interpretative voice attributes by means of minimizing adversarial loss to make the encoded information mutually independent while preserving the capacity to generate a speech signal from the disentangled codes. During inference for voice conversion, the disentangled voice attributes can be manipulated and the speech signal can be generated accordingly. For experimental evaluation, the proposed method is applied to the task of voice gender conversion using the freely available VCTK dataset. Quantitative measurements of mutual information between the variables of speaker identity and speaker gender show that the proposed architecture can learn gender-independent representation of speakers. Additional measurements of speaker recognition indicate that speaker identity can be recognized accurately from the gender-independent representation. Finally, a subjective experiment conducted on the task of voice gender manipulation shows that the proposed architecture can convert voice gender with very high efficiency and good naturalness.
Open AccessManipulating Voice Attributes by Adversarial Learning of Structured Disentangled Representations
Voice conversion (VC) consists of digitally altering the voice of an individual to manipulate part of its content, primarily its identity, while maintaining the rest unchanged. Research in neural VC has accomplished considerable breakthroughs with the capacity to falsify a voice identity using a small amount of data with a highly realistic rendering. This paper goes beyond voice identity manipulation and presents an original neural architecture that allows the manipulation of voice attributes (e.g., gender and age). The proposed architecture is inspired by the fader network, transferring the same ideas to voice manipulation. The information conveyed by the speech signal is disentangled into interpretative voice attributes by means of minimizing adversarial loss to make the encoded information mutually independent while preserving the capacity to generate a speech signal from the disentangled codes. During inference for voice conversion, the disentangled voice attributes can be manipulated and the speech signal can be generated accordingly. For experimental evaluation, the proposed method is applied to the task of voice gender conversion using the freely available VCTK dataset. Quantitative measurements of mutual information between the variables of speaker identity and speaker gender show that the proposed architecture can learn gender-independent representation of speakers. Additional measurements of speaker recognition indicate that speaker identity can be recognized accurately from the gender-independent representation. Finally, a subjective experiment conducted on the task of voice gender manipulation shows that the proposed architecture can convert voice gender with very high efficiency and good naturalness.
- Open Access
- 31 January 2023
- Published by MDPI AG in Applied Sciences
- Vol. 13 (3) , 1854
A Linear Memory CTC-Based Algorithm for Text-to-Voice Alignment of Very Long Audio Recordings
Synchronisation of a voice recording with the corresponding text is a common task in speech and music processing, and is used in many practical applications (automatic subtitling, audio indexing, etc.). A common approach derives a mid-level feature from the audio and finds its alignment to the text by means of maximizing a similarity measure via Dynamic Time Warping (DTW). Recently, a Connectionist Temporal Classification (CTC) approach was proposed that directly emits character probabilities and uses those to find the optimal text-to-voice alignment. While this method yields promising results, the memory complexity of the optimal alignment search remains quadratic in input lengths, limiting its application to relatively short recordings. In this work, we describe how recent improvements brought to the textbook DTW algorithm can be adapted to the CTC context to achieve linear memory complexity. We then detail our overall solution and demonstrate that it can align text to several hours of audio with a mean alignment error of 50 ms for speech, and 120 ms for singing voice, which corresponds to a median alignment error that is below 50 ms for both voice types. Finally, we evaluate its robustness to transcription errors and different languages.
- 31 January 2023
- Published by MDPI AG in Applied Sciences
- Vol. 13 (3) , 1854
Open AccessA Linear Memory CTC-Based Algorithm for Text-to-Voice Alignment of Very Long Audio Recordings
Synchronisation of a voice recording with the corresponding text is a common task in speech and music processing, and is used in many practical applications (automatic subtitling, audio indexing, etc.). A common approach derives a mid-level feature from the audio and finds its alignment to the text by means of maximizing a similarity measure via Dynamic Time Warping (DTW). Recently, a Connectionist Temporal Classification (CTC) approach was proposed that directly emits character probabilities and uses those to find the optimal text-to-voice alignment. While this method yields promising results, the memory complexity of the optimal alignment search remains quadratic in input lengths, limiting its application to relatively short recordings. In this work, we describe how recent improvements brought to the textbook DTW algorithm can be adapted to the CTC context to achieve linear memory complexity. We then detail our overall solution and demonstrate that it can align text to several hours of audio with a mean alignment error of 50 ms for speech, and 120 ms for singing voice, which corresponds to a median alignment error that is below 50 ms for both voice types. Finally, we evaluate its robustness to transcription errors and different languages.
- Open Access
- 23 February 2022
- Published by MDPI AG in Information
- Vol. 13 (3) , 103
Neural Vocoding for Singing and Speaking Voices with the Multi-Band Excited WaveNet
The use of the mel spectrogram as a signal parameterization for voice generation is quite recent and linked to the development of neural vocoders. These are deep neural networks that allow reconstructing high-quality speech from a given mel spectrogram. While initially developed for speech synthesis, now neural vocoders have also been studied in the context of voice attribute manipulation, opening new means for voice processing in audio production. However, to be able to apply neural vocoders in real-world applications, two problems need to be addressed: (1) To support use in professional audio workstations, the computational complexity should be small, (2) the vocoder needs to support a large variety of speakers, differences in voice qualities, and a wide range of intensities potentially encountered during audio production. In this context, the present study will provide a detailed description of the Multi-band Excited WaveNet, a fully convolutional neural vocoder built around signal processing blocks. It will evaluate the performance of the vocoder when trained on a variety of multi-speaker and multi-singer databases, including an experimental evaluation of the neural vocoder trained on speech and singing voices. Addressing the problem of intensity variation, the study will introduce a new adaptive signal normalization scheme that allows for robust compensation for dynamic and static gain variations. Evaluations are performed using objective measures and a number of perceptual tests including different neural vocoder algorithms known from the literature. The results confirm that the proposed vocoder compares favorably to the state-of-the-art in its capacity to generalize to unseen voices and voice qualities. The remaining challenges will be discussed.
- 23 February 2022
- Published by MDPI AG in Information
- Vol. 13 (3) , 103
Open AccessNeural Vocoding for Singing and Speaking Voices with the Multi-Band Excited WaveNet
The use of the mel spectrogram as a signal parameterization for voice generation is quite recent and linked to the development of neural vocoders. These are deep neural networks that allow reconstructing high-quality speech from a given mel spectrogram. While initially developed for speech synthesis, now neural vocoders have also been studied in the context of voice attribute manipulation, opening new means for voice processing in audio production. However, to be able to apply neural vocoders in real-world applications, two problems need to be addressed: (1) To support use in professional audio workstations, the computational complexity should be small, (2) the vocoder needs to support a large variety of speakers, differences in voice qualities, and a wide range of intensities potentially encountered during audio production. In this context, the present study will provide a detailed description of the Multi-band Excited WaveNet, a fully convolutional neural vocoder built around signal processing blocks. It will evaluate the performance of the vocoder when trained on a variety of multi-speaker and multi-singer databases, including an experimental evaluation of the neural vocoder trained on speech and singing voices. Addressing the problem of intensity variation, the study will introduce a new adaptive signal normalization scheme that allows for robust compensation for dynamic and static gain variations. Evaluations are performed using objective measures and a number of perceptual tests including different neural vocoder algorithms known from the literature. The results confirm that the proposed vocoder compares favorably to the state-of-the-art in its capacity to generalize to unseen voices and voice qualities. The remaining challenges will be discussed.
- Open Access
- 23 February 2022
- Published by MDPI AG in Information
- Vol. 13 (3) , 102
A Bottleneck Auto-Encoder for F0 Transformations on Speech and Singing Voice
In this publication, we present a deep learning-based method to transform the in speech and singing voice recordings. transformation is performed by training an auto-encoder on the voice signal’s mel-spectrogram and conditioning the auto-encoder on the . Inspired by AutoVC/F0, we apply an information bottleneck to it to disentangle the from its latent code. The resulting model successfully applies the desired to the input mel-spectrograms and adapts the speaker identity when necessary, e.g., if the requested falls out of the range of the source speaker/singer. Using the mean error in the transformed mel-spectrograms, we define a disentanglement measure and perform a study over the required bottleneck size. The study reveals that to remove the from the auto-encoder’s latent code, the bottleneck size should be smaller than four for singing and smaller than nine for speech. Through a perceptive test, we compare the audio quality of the proposed auto-encoder to transformations obtained with a classical vocoder. The perceptive test confirms that the audio quality is better for the auto-encoder than for the classical vocoder. Finally, a visual analysis of the latent code for the two-dimensional case is carried out. We observe that the auto-encoder encodes phonemes as repeated discontinuous temporal gestures within the latent code.- 23 February 2022
- Published by MDPI AG in Information
- Vol. 13 (3) , 102
Open AccessA Bottleneck Auto-Encoder for F0 Transformations on Speech and Singing Voice
In this publication, we present a deep learning-based method to transform the in speech and singing voice recordings. transformation is performed by training an auto-encoder on the voice signal’s mel-spectrogram and conditioning the auto-encoder on the . Inspired by AutoVC/F0, we apply an information bottleneck to it to disentangle the from its latent code. The resulting model successfully applies the desired to the input mel-spectrograms and adapts the speaker identity when necessary, e.g., if the requested falls out of the range of the source speaker/singer. Using the mean error in the transformed mel-spectrograms, we define a disentanglement measure and perform a study over the required bottleneck size. The study reveals that to remove the from the auto-encoder’s latent code, the bottleneck size should be smaller than four for singing and smaller than nine for speech. Through a perceptive test, we compare the audio quality of the proposed auto-encoder to transformations obtained with a classical vocoder. The perceptive test confirms that the audio quality is better for the auto-encoder than for the classical vocoder. Finally, a visual analysis of the latent code for the two-dimensional case is carried out. We observe that the auto-encoder encodes phonemes as repeated discontinuous temporal gestures within the latent code. - Open Access
CycleGAN Voice Conversion of Spectral Envelopes using Adversarial Weights
This paper tackles GAN optimization and stability issues in the context of voice conversion. First, to simplify the conversion task, we propose to use spectral envelopes as inputs. Second we propose two adversarial weight training paradigms, the generalized weighted GAN and the generator impact GAN, both aim at reducing the impact of the generator on the discriminator, so both can learn more gradually and efficiently during training. Applying an energy constraint to the cycleGAN paradigm considerably improved conversion quality. A subjective experiment conducted on a voice conversion task on the voice conversion challenge 2018 dataset shows first that despite a significantly reduced network complexity, the proposed method achieves state-of-the-art results, and second that the proposed weighted GAN methods outperform a previously proposed one.
Open AccessCycleGAN Voice Conversion of Spectral Envelopes using Adversarial Weights
This paper tackles GAN optimization and stability issues in the context of voice conversion. First, to simplify the conversion task, we propose to use spectral envelopes as inputs. Second we propose two adversarial weight training paradigms, the generalized weighted GAN and the generator impact GAN, both aim at reducing the impact of the generator on the discriminator, so both can learn more gradually and efficiently during training. Applying an energy constraint to the cycleGAN paradigm considerably improved conversion quality. A subjective experiment conducted on a voice conversion task on the voice conversion challenge 2018 dataset shows first that despite a significantly reduced network complexity, the proposed method achieves state-of-the-art results, and second that the proposed weighted GAN methods outperform a previously proposed one.
- Open Access
Semi-supervised learning of glottal pulse positions in a neural analysis-synthesis framework
This article investigates into recently emerging approaches that use deep neural networks for the estimation of glottal closure instants (GCI). We build upon our previous approach that used synthetic speech exclusively to create perfectly annotated training data and that had been shown to compare favourably with other training approaches using electroglottograph (EGG) signals. Here we introduce a semi-supervised training strategy that allows refining the estimator by means of an analysis-synthesis setup using real speech signals, for which GCI ground truth does not exist. Evaluation of the analyser is performed by means of comparing the GCI extracted from the glottal flow signal generated by the analyser with the GCI extracted from EGG on the CMU arctic dataset, where EGG signals were recorded in addition to speech. We observe that (1.) the artificial increase of the diversity of pulse shapes that has been used in our previous construction of the synthetic database is beneficial, (2.) training the GCI network in the analysis-synthesis setup allows achieving a very significant improvement of the GCI analyser, (3.) additional regularisation strategies allow improving the final analysis network when trained in the analysis-synthesis setup.
Open AccessSemi-supervised learning of glottal pulse positions in a neural analysis-synthesis framework
This article investigates into recently emerging approaches that use deep neural networks for the estimation of glottal closure instants (GCI). We build upon our previous approach that used synthetic speech exclusively to create perfectly annotated training data and that had been shown to compare favourably with other training approaches using electroglottograph (EGG) signals. Here we introduce a semi-supervised training strategy that allows refining the estimator by means of an analysis-synthesis setup using real speech signals, for which GCI ground truth does not exist. Evaluation of the analyser is performed by means of comparing the GCI extracted from the glottal flow signal generated by the analyser with the GCI extracted from EGG on the CMU arctic dataset, where EGG signals were recorded in addition to speech. We observe that (1.) the artificial increase of the diversity of pulse shapes that has been used in our previous construction of the synthetic database is beneficial, (2.) training the GCI network in the analysis-synthesis setup allows achieving a very significant improvement of the GCI analyser, (3.) additional regularisation strategies allow improving the final analysis network when trained in the analysis-synthesis setup.
- Open Access
- 1 January 2020
- Published by OpenEdition in Volume !
- Vol. 16 : 2 / 1 (16 : 2 / 1) , 63-85
Analyse du style vocal et modélisation pour la synthèse de chant expressif : l’exemple d’Édith Piaf
Un nombre croissant de musicologues s’intéresse aujourd’hui aux outils d’analyse spectrale pour étudier l’interprétation vocale. Permettant une visualisation acoustique du son, ces outils rendent possibles l’analyse et l’objectivation de caractéristiques interprétatives longtemps considérées comme échappant à la théorisation : variations agogiques du rythme ou de la mélodie, timbre… Cependant, ce type d’échange entre musicologie et sciences exactes est souvent à sens unique : le musicologue exploite des logiciels à des fins musicologiques. L’expérience que nous allons présenter a la particularité de mettre en œuvre une collaboration simultanée entre chercheurs en traitement du signal et musicologues autour d’un même objectif : l’élaboration d’un système de synthèse du chant pouvant reproduire la diversité des « styles vocaux » interprétatifs. Elle se place à la convergence de deux champs de recherche autour de la voix chantée : recherches en informatique et traitement de signal sur la modélisation et la synthèse de la voix chantée, recherches en musicologie sur l’interprétation vocale et les styles interprétatifs dans la chanson populaire francophone. Cette approche donne, à travers la modélisation du style vocal d’Édith Piaf, un exemple de la manière dont deux disciplines peuvent collaborer fructueusement pour s’informer et s’enrichir mutuellement.
- 1 January 2020
- Published by OpenEdition in Volume !
- Vol. 16 : 2 / 1 (16 : 2 / 1) , 63-85
Open AccessAnalyse du style vocal et modélisation pour la synthèse de chant expressif : l’exemple d’Édith Piaf
Un nombre croissant de musicologues s’intéresse aujourd’hui aux outils d’analyse spectrale pour étudier l’interprétation vocale. Permettant une visualisation acoustique du son, ces outils rendent possibles l’analyse et l’objectivation de caractéristiques interprétatives longtemps considérées comme échappant à la théorisation : variations agogiques du rythme ou de la mélodie, timbre… Cependant, ce type d’échange entre musicologie et sciences exactes est souvent à sens unique : le musicologue exploite des logiciels à des fins musicologiques. L’expérience que nous allons présenter a la particularité de mettre en œuvre une collaboration simultanée entre chercheurs en traitement du signal et musicologues autour d’un même objectif : l’élaboration d’un système de synthèse du chant pouvant reproduire la diversité des « styles vocaux » interprétatifs. Elle se place à la convergence de deux champs de recherche autour de la voix chantée : recherches en informatique et traitement de signal sur la modélisation et la synthèse de la voix chantée, recherches en musicologie sur l’interprétation vocale et les styles interprétatifs dans la chanson populaire francophone. Cette approche donne, à travers la modélisation du style vocal d’Édith Piaf, un exemple de la manière dont deux disciplines peuvent collaborer fructueusement pour s’informer et s’enrichir mutuellement.