Keelan Evanini
Current Organizations / Institutes
The organizations/institutes from the scholar's latest article
Keelan Evanini has a total of 53 co-authors. Here are the top co-authors by number of shared publications:
- Vikram Ramanarayanan
Modality.AI, Inc., San Francisco, CA
4 shared publications Xinhao Wang
Educational Testing Service Princeton NJ
4 shared publications- Xinhao Wang
Wangxuan Institute of Computer Technology, Peking University, Beijing, China
4 shared publications David Suendermann-Oeft
3 shared publications- Klaus Zechner
Educational Testing Service Princeton NJ USA
3 shared publications Jidong Tao
Educational Testing Service Princeton NJ
2 shared publications
Latest Publications
- Open Access
Do face masks introduce bias in speech technologies? The case of automated scoring of speaking proficiency
The COVID-19 pandemic has led to a dramatic increase in the use of face masks worldwide. Face coverings can affect both acoustic properties of the signal as well as speech patterns and have unintended effects if the person wearing the mask attempts to use speech processing technologies. In this paper we explore the impact of wearing face masks on the automated assessment of English language proficiency. We use a dataset from a large-scale speaking test for which test-takers were required to wear face masks during the test administration, and we compare it to a matched control sample of test-takers who took the same test before the mask requirements were put in place. We find that the two samples differ across a range of acoustic measures and also show a small but significant difference in speech patterns. However, these differences do not lead to differences in human or automated scores of English language proficiency. Several measures of bias showed no differences in scores between the two groups.
Open AccessDo face masks introduce bias in speech technologies? The case of automated scoring of speaking proficiency
The COVID-19 pandemic has led to a dramatic increase in the use of face masks worldwide. Face coverings can affect both acoustic properties of the signal as well as speech patterns and have unintended effects if the person wearing the mask attempts to use speech processing technologies. In this paper we explore the impact of wearing face masks on the automated assessment of English language proficiency. We use a dataset from a large-scale speaking test for which test-takers were required to wear face masks during the test administration, and we compare it to a matched control sample of test-takers who took the same test before the mask requirements were put in place. We find that the two samples differ across a range of acoustic measures and also show a small but significant difference in speech patterns. However, these differences do not lead to differences in human or automated scores of English language proficiency. Several measures of bias showed no differences in scores between the two groups.
- 1 December 2019
- Published by Institute of Electrical and Electronics Engineers (IEEE)
- p. 764-771
Using Very Deep Convolutional Neural Networks to Automatically Detect Plagiarized Spoken Responses
This study focuses on the automatic plagiarism detection in the context of high-stakes spoken language proficiency assessment, in which some test takers may attempt to game the test by memorizing prepared source materials before the test and then adapting them on-the-fly during the test to produce their spoken responses. When trying to identify such instances of plagiarism, experienced human raters attempt to find salient matching expressions that appear both in potential source materials and the test responses. This motivates an approach that visualizes a grid of lexical matches between a test response and a source and then applies state-of-the-art image recognition techniques to detect patterns of matching sequences. This study employs Inception networks-very deep convolutional neural networks-to build automatic detection models. The system achieves an F1-score of 79.6% on the class of plagiarized responses outperforming a baseline system based on word sequence matching (F1-score of 74.1%).
- 1 December 2019
- Published by Institute of Electrical and Electronics Engineers (IEEE)
- p. 764-771
Using Very Deep Convolutional Neural Networks to Automatically Detect Plagiarized Spoken Responses
This study focuses on the automatic plagiarism detection in the context of high-stakes spoken language proficiency assessment, in which some test takers may attempt to game the test by memorizing prepared source materials before the test and then adapting them on-the-fly during the test to produce their spoken responses. When trying to identify such instances of plagiarism, experienced human raters attempt to find salient matching expressions that appear both in potential source materials and the test responses. This motivates an approach that visualizes a grid of lexical matches between a test response and a source and then applies state-of-the-art image recognition techniques to detect patterns of matching sequences. This study employs Inception networks-very deep convolutional neural networks-to build automatic detection models. The system achieves an F1-score of 79.6% on the class of plagiarized responses outperforming a baseline system based on word sequence matching (F1-score of 74.1%).
- 28 November 2019
- Published by Taylor & Francis
- p. 61-74
Automatic Speech Recognition for Automated Speech Scoring
Automatic speech scoring systems rely on accurate Automatic Speech Recognition (ASR) for optimal performance; therefore, in order to accurately describe the strengths and limitations of automatic speech scoring systems, this chapter presents information about this crucial component in the system’s pipeline. ASR research is a multidisciplinary area that incorporates knowledge from the fields of electrical engineering, computer science, mathematics, and linguistics. Research on ASR began in the late 1940s with support from the US Department of Defense. In 1952, the first spoken digit recognizer for a single speaker was built at Bell Laboratories. ASR is the computational process that converts a speech signal contained in a digital audio recording to a text output. The speech signal captured by a microphone can be converted into digital form by an analog-to-digital converter and stored in the form of a waveform, which is a representation of how a signal changes over time.
- 28 November 2019
- Published by Taylor & Francis
- p. 61-74
Automatic Speech Recognition for Automated Speech Scoring
Automatic speech scoring systems rely on accurate Automatic Speech Recognition (ASR) for optimal performance; therefore, in order to accurately describe the strengths and limitations of automatic speech scoring systems, this chapter presents information about this crucial component in the system’s pipeline. ASR research is a multidisciplinary area that incorporates knowledge from the fields of electrical engineering, computer science, mathematics, and linguistics. Research on ASR began in the late 1940s with support from the US Department of Defense. In 1952, the first spoken digit recognizer for a single speaker was built at Bell Laboratories. ASR is the computational process that converts a speech signal contained in a digital audio recording to a text output. The speech signal captured by a microphone can be converted into digital form by an analog-to-digital converter and stored in the form of a waveform, which is a representation of how a signal changes over time.
- 23 November 2018
- Published by Taylor & Francis in Computer Assisted Language Learning
- Vol. 32 (4) , 398-417
Evaluating English language learners’ conversations: Man vs. Machine
This study investigated a new conversation-based assessment for English language learners. In this assessment, students converse with animated agents in natural language conversations to assess various aspects of English language learning. In a between subjects design, 31 students (N = 31) were asked questions by either the newly created system or human interviewers. Results revealed that there was no difference in answers in regards to impacting the constructs under investigation. However, there were other differences including higher word count and potentially more complex language usage when speaking to a human. Furthermore, students liked interacting with the system. These results suggest that the newly created system can assess students’ English capabilities comparably to human interviewers and implementing such a formative assessment is feasible.
- 23 November 2018
- Published by Taylor & Francis in Computer Assisted Language Learning
- Vol. 32 (4) , 398-417
Evaluating English language learners’ conversations: Man vs. Machine
This study investigated a new conversation-based assessment for English language learners. In this assessment, students converse with animated agents in natural language conversations to assess various aspects of English language learning. In a between subjects design, 31 students (N = 31) were asked questions by either the newly created system or human interviewers. Results revealed that there was no difference in answers in regards to impacting the constructs under investigation. However, there were other differences including higher word count and potentially more complex language usage when speaking to a human. Furthermore, students liked interacting with the system. These results suggest that the newly created system can assess students’ English capabilities comparably to human interviewers and implementing such a formative assessment is feasible.
- Open Access
- 7 May 2018
- Published by Taylor & Francis in Computer Assisted Language Learning
- Vol. 32 (1-2) , 17-48
Simulated speaking environments for language learning: insights from three cases
Recent CALL technology reviews cover a plethora of technologies available to language learners to improve a variety of skills, including speaking. However, few technology-enhanced self-access tools are available for pragmatic development, especially in oral modality. Recognizing the benefits of structured practice for second language development, we demonstrate how such practice can be incorporated into three recently developed simulated speaking environments that vary on the targeted L2 (French, English), domain of use (academic or everyday interaction), emphasis on higher-order and/or lower-order skills, and accommodation of multiple L2 varieties. In the spirit of finding synergies and learning from each other's experiences in specific local contexts, we address the following research questions: (1) How does the local context, researcher and learner goals, and technological possibilities influence the design of each computer application? (2) Based on the examination of the three programs, what can we learn in view of redesign options and suggest to future developers of such programs?
- 7 May 2018
- Published by Taylor & Francis in Computer Assisted Language Learning
- Vol. 32 (1-2) , 17-48
Open AccessSimulated speaking environments for language learning: insights from three cases
Recent CALL technology reviews cover a plethora of technologies available to language learners to improve a variety of skills, including speaking. However, few technology-enhanced self-access tools are available for pragmatic development, especially in oral modality. Recognizing the benefits of structured practice for second language development, we demonstrate how such practice can be incorporated into three recently developed simulated speaking environments that vary on the targeted L2 (French, English), domain of use (academic or everyday interaction), emphasis on higher-order and/or lower-order skills, and accommodation of multiple L2 varieties. In the spirit of finding synergies and learning from each other's experiences in specific local contexts, we address the following research questions: (1) How does the local context, researcher and learner goals, and technological possibilities influence the design of each computer application? (2) Based on the examination of the three programs, what can we learn in view of redesign options and suggest to future developers of such programs?
- Open Access
- 12 April 2018
- Published by Wiley in ETS Research Report Series
- Vol. 2018 (1) , 1-31
Automated Scoring of Nonnative Speech Using the SpeechRaterSMv. 5.0 Engine
This research report provides an overview of the R&D efforts at Educational Testing Service related to its capability for automated scoring of nonnative spontaneous speech with theSpeechRaterSMautomated scoring service since its initial version was deployed in 2006. While most aspects of this R&D work have been published in various venues in recent years, no comprehensive account of the current state of SpeechRater has been provided since the initial publications following its first operational use in 2006. After a brief review of recent related work by other institutions, we summarize the main features and feature classes that have been developed and introduced into SpeechRater in the past 10 years, including features measuring aspects of pronunciation, prosody, vocabulary, grammar, content, and discourse. Furthermore, new types of filtering models for flagging nonscorable spoken responses are described, as is our new hybrid way of building linear regression scoring models with improved feature selection. Finally, empirical results for SpeechRater 5.0 (operationally deployed in 2016) are provided.
- 12 April 2018
- Published by Wiley in ETS Research Report Series
- Vol. 2018 (1) , 1-31
Open AccessAutomated Scoring of Nonnative Speech Using the SpeechRaterSMv. 5.0 Engine
This research report provides an overview of the R&D efforts at Educational Testing Service related to its capability for automated scoring of nonnative spontaneous speech with theSpeechRaterSMautomated scoring service since its initial version was deployed in 2006. While most aspects of this R&D work have been published in various venues in recent years, no comprehensive account of the current state of SpeechRater has been provided since the initial publications following its first operational use in 2006. After a brief review of recent related work by other institutions, we summarize the main features and feature classes that have been developed and introduced into SpeechRater in the past 10 years, including features measuring aspects of pronunciation, prosody, vocabulary, grammar, content, and discourse. Furthermore, new types of filtering models for flagging nonscorable spoken responses are described, as is our new hybrid way of building linear regression scoring models with improved feature selection. Finally, empirical results for SpeechRater 5.0 (operationally deployed in 2016) are provided.
- 20 August 2017
- Published by International Speech Communication Association
- p. 2586-2590
Improving Sub-Phone Modeling for Better Native Language Identification with Non-Native English Speech
- 20 August 2017
- Published by International Speech Communication Association
- p. 2586-2590
Improving Sub-Phone Modeling for Better Native Language Identification with Non-Native English Speech
- 2 May 2017
- Published by Wiley in ETS Research Report Series
- Vol. 2017 (1) , 1-11
Approaches to Automated Scoring of Speaking for K–12 English Language Proficiency Assessments
This report is the fifth in a series concerning English language proficiency (ELP) assessments for English learners (ELs) in kindergarten through 12th grade in the United States. The series, produced by Educational Testing Service (ETS), is intended to provide theory‐ and evidence‐based principles and recommendations for improving next‐generationELPassessment systems, policies, and practices and to stimulate discussion on better serving K–12ELstudents. The first report articulated a high‐level conceptualization of next‐generationELPassessment systems (Hauck, Wolf, & Mislevy, 2016). The second report addressed accessibility issues in the context ofELPassessments forELsandELswith disabilities (Guzman‐Orth, Laitusis, Thurlow, & Christensen, 2016). The third report focused on critical policy and research issues of summativeELPassessments that state use for accountability purposes (Wolf, Guzman‐Orth, & Hauck, 2016). The fourth report dealt with one of the major uses ofELPassessments—the initial identification and classification of ELs (Lopez, Pooler, & Linquanti, 2016). The present report discusses approaches to using automated scoring technology for the evaluation of student‐spoken responses on K–12ELPassessments. As many states have begun to use computer‐basedELPassessments, there is a growing interest in automated scoring of spoken responses to increase the efficiency of scoring. This report delineates major areas to consider in using automated speech scoring for K–12ELPassessments (i.e., assessment construct and task design, scoring and score reporting, and artificial intelligence (AI) model development and test delivery) and makes recommendations for states on how to evaluate these considerations and determine a path forward.
- 2 May 2017
- Published by Wiley in ETS Research Report Series
- Vol. 2017 (1) , 1-11
Approaches to Automated Scoring of Speaking for K–12 English Language Proficiency Assessments
This report is the fifth in a series concerning English language proficiency (ELP) assessments for English learners (ELs) in kindergarten through 12th grade in the United States. The series, produced by Educational Testing Service (ETS), is intended to provide theory‐ and evidence‐based principles and recommendations for improving next‐generationELPassessment systems, policies, and practices and to stimulate discussion on better serving K–12ELstudents. The first report articulated a high‐level conceptualization of next‐generationELPassessment systems (Hauck, Wolf, & Mislevy, 2016). The second report addressed accessibility issues in the context ofELPassessments forELsandELswith disabilities (Guzman‐Orth, Laitusis, Thurlow, & Christensen, 2016). The third report focused on critical policy and research issues of summativeELPassessments that state use for accountability purposes (Wolf, Guzman‐Orth, & Hauck, 2016). The fourth report dealt with one of the major uses ofELPassessments—the initial identification and classification of ELs (Lopez, Pooler, & Linquanti, 2016). The present report discusses approaches to using automated scoring technology for the evaluation of student‐spoken responses on K–12ELPassessments. As many states have begun to use computer‐basedELPassessments, there is a growing interest in automated scoring of spoken responses to increase the efficiency of scoring. This report delineates major areas to consider in using automated speech scoring for K–12ELPassessments (i.e., assessment construct and task design, scoring and score reporting, and artificial intelligence (AI) model development and test delivery) and makes recommendations for states on how to evaluate these considerations and determine a path forward.
- 2 May 2017
- Published by Wiley in ETS Research Report Series
- Vol. 2017 (1) , 1-7
A Multimodal Dialog System for Language Assessment: Current State and Future Directions
- 2 May 2017
- Published by Wiley in ETS Research Report Series
- Vol. 2017 (1) , 1-7
A Multimodal Dialog System for Language Assessment: Current State and Future Directions