The influence of automatic speech recognition accuracy on the performance of an automated speech assessment system
- 1 December 2014
- conference paper
- Published by Institute of Electrical and Electronics Engineers (IEEE)
- p. 294-299
Abstract
The effectiveness of automated scoring systems for evaluating spoken language proficiency depends greatly on the quality of the automatic speech recognition (ASR) output that is used to calculate the features for the scoring model. In this paper, we examine the effects of ASR word error rate (WER) on the scores produced by a system for automated scoring of non-native English speaking proficiency, as well as on the scoring model features (especially content features) in order to demonstrate the impact of ASR improvements on the performance of the automated speech assessment system. Five different sets of transcriptions with varying degrees of WER ranging from 0% to 52% (including four sets of ASR hypotheses and manual transcriptions) were obtained for a dataset of spoken responses from a pilot administration of an assessment of non-native English speaking proficiency. The experimental results show that higher performing ASR leads to better performance in the automated assessment system; furthermore, the correlation between human and automated scores drops substantially with an increase in WER from 10.7% to 28.9%, whereas the correlation changes little within the following two ranges of WERs: 0% to 10.7% and 28.9% to 52%. A detailed analysis of the features used in the scoring model shows that the ASR errors have a bigger impact on the content features than the delivery and language use features.Keywords
This publication has 8 references indexed in Scilit:
- Evaluation of disfluent speech by means of automatic acoustic measurementsThe Journal of the Acoustical Society of America, 2014
- Automated scoring of speaking items in an assessment for teachers of English as a Foreign Language2014
- Human and automated assessment of oral reading fluency.Journal of Educational Psychology, 2013
- Deep Neural Networks for Acoustic Modeling in Speech Recognition: The Shared Views of Four Research GroupsIEEE Signal Processing Magazine, 2012
- Automatic scoring of non-native spontaneous speech in tests of spoken EnglishSpeech Communication, 2009
- Quantitative assessment of second language learners’ fluency: Comparisons between read and spontaneous speechThe Journal of the Acoustical Society of America, 2002
- Quantitative assessment of second language learners’ fluency by means of automatic speech recognition technologyThe Journal of the Acoustical Society of America, 2000
- Phone-level pronunciation scoring and assessment for interactive language learningSpeech Communication, 2000