Multi-Stream Acoustic Modelling Using Raw Real and Imaginary Parts of the Fourier Transform
- 26 January 2023
- journal article
- research article
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE/ACM Transactions on Audio, Speech, and Language Processing
- Vol. 31 (23299290) , 876-890
Abstract
In this paper, we investigate multi-stream acoustic modelling using the raw real and imaginary parts of the Fourier transform of speech signals. Using the raw magnitude spectrum, or features derived from it, as a proxy for the real and imaginary parts leads to irreversible information loss and suboptimal information fusion. We discuss and quantify the importance of such information in terms of speech quality and intelligibility. In the proposed framework, the real and imaginary parts are treated as two streams of information, pre-processed via separate convolutional networks, and then combined at an optimal level of abstraction, followed by further post-processing via recurrent and fully-connected layers. The optimal level of information fusion in various architectures, training dynamics in terms of cross-entropy loss, frame classification accuracy and WER as well as the shape and properties of the filters learned in the first convolutional layer of single- and multi-stream models are analysed. We investigated the effectiveness of the proposed systems in various tasks: TIMIT/NTIMIT (phone recognition), Aurora-4 (noise robustness), WSJ (read speech), AMI (meeting) and TORGO (dysarthric speech). Across all tasks we achieved competitive performance: in Aurora-4, down to 4.6% WER on average, in WSJ down to 4.6% and 6.2% WERs for Eval-92 and Eval-93, for Dev/Eval sets of the AMI-IHM down to 23.3%/23.8% WERs and in the AMI-SDM down to 43.7%/47.6% WERs have been achieved. In TORGO, for dysarthric and typical speech we achieved down to 31.7% and 10.2% WERs, respectively.Keywords
Funding Information
- EPSRC project (EP/R012180/1)
This publication has 64 references indexed in Scilit:
- The TORGO database of acoustic and articulatory speech from speakers with dysarthriaLanguage Resources and Evaluation, 2011
- Subjective comparison and evaluation of speech enhancement algorithmsSpeech Communication, 2007
- On the usefulness of STFT phase spectrum in human listening testsSpeech Communication, 2005
- Likelihood-Maximizing Beamforming for Robust Hands-Free Speech RecognitionIEEE Transactions on Speech and Audio Processing, 2004
- Perceptual linear predictive (PLP) analysis of speechThe Journal of the Acoustical Society of America, 1990
- Spectral analysis using generalized cepstrumIEEE Transactions on Acoustics, Speech, and Signal Processing, 1984
- Signal estimation from modified short-time Fourier transformIEEE Transactions on Acoustics, Speech, and Signal Processing, 1984
- Signal reconstruction from signed Fourier transform magnitudeIEEE Transactions on Acoustics, Speech, and Signal Processing, 1983
- Signal reconstruction from phase or magnitudeIEEE Transactions on Acoustics, Speech, and Signal Processing, 1980
- Comparison of parametric representations for monosyllabic word recognition in continuously spoken sentencesIEEE Transactions on Acoustics, Speech, and Signal Processing, 1980