Multi-Feature and Multi-Modal Mispronunciation Detection and Diagnosis Method Based on the Squeezeformer Encoder
Open Access
- 22 May 2023
- journal article
- research article
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Access
- Vol. 11, 66245-66256
Abstract
In recent years, with the development of deep learning, research on end-to-end mispronunciation detection and diagnosis(MDD) methods has been further promoted. At present, research on end-to-end mispronunciation detection and diagnosis is gradually emerging. Most end-to-end mispronunciation detection and diagnosis methods are based on the CNN-RNN-CTC network structure. To improve the performance of end-to-end mispronunciation detection and diagnosis systems, this paper proposes an end-to-end multi-feature and multi-modal mispronunciation detection and diagnosis method based on the Squeezeformer encoder. The model uses Squeezeformer as an audio encoder, a Bi-LSTM network as a phoneme encoder, and Transformer as a decoder. The model fuses phoneme information before speech encoding and decoding, respectively, and uses a secondary decoding mechanism during the decoding process. This study further incorporated phoneme information in the encoding process so that the model could learn the intrinsic characteristics of the speaker’s pronunciation content. The decoding process uses a secondary decoding mechanism to send the sequence decoded by the model to the decoder for decoding again, which solves the problem of no a priori knowledge at the decoder end in the first decoding stage, thus improving the performance of mispronunciation detection and diagnosis. In this study, experiments were conducted on the PSC-Reading Mandarin mispronunciation detection and diagnosis dataset. Compared with the baseline model, the F1 index improved from 0.4060 to 0.7943, and the diagnostic accuracy improved from 83.93% to 88.45%.Keywords
Funding Information
- National Natural Science Foundation Project (U1903213)
This publication has 30 references indexed in Scilit:
- Hybrid CTC/Attention Architecture for End-to-End Speech RecognitionIEEE Journal of Selected Topics in Signal Processing, 2017
- Convolution in Convolution for Network in NetworkIEEE Transactions on Neural Networks and Learning Systems, 2017
- Mispronunciation Detection and Diagnosis in L2 English Speech Using Multidistribution Deep Neural NetworksIEEE/ACM Transactions on Audio, Speech, and Language Processing, 2016
- Improved mispronunciation detection with deep neural network trained acoustic models and transfer learning based logistic regression classifiersSpeech Communication, 2015
- Beam search algorithms for multilabel learningMachine Learning, 2013
- Long Short-Term Memory2012
- GMM-HMM acoustic model training by a two level procedure with Gaussian components determined by automatic model selection2010
- Confidence measures for large vocabulary continuous speech recognitionIEEE Transactions on Speech and Audio Processing, 2001
- A global optimization algorithm (GOP) for certain classes of nonconvex NLPs—I. TheoryComputers & Chemical Engineering, 1990
- The viterbi algorithmProceedings of the IEEE, 1973