Yasin Karatay
Yasin Karatay has a total of 17 co-authors. Here are the top co-authors by number of shared publications:
- Jing Xu
Cambridge University Press & Assessment, UK
3 shared publications - Ananda Astrini Muhammad
Iowa State University, Ames, Iowa, USA
2 shared publications - Haeyun Jin
Iowa State University, United States
2 shared publications - Timothy Kochem
Iowa State University, United States
2 shared publications - Carole A. Chapelle
Iowa State University, USA
1 shared publication - Carsten Roever
University of Melbourne
1 shared publication
Latest Publications
- 30 July 2026
- Published by SAGE Publications in Language Testing
- Vol. 43 (4) , 424-456
Evaluating the Validity of a Spoken Dialog System-Mediated Role-Play Test for Assessing Second Language Oral Communication
Generative AI offers new opportunities for interactive oral communication assessment for specific purposes by building on past research investigating test takers’ oral interaction with spoken dialog systems (SDSs). Performing as automated conversational agents, SDSs can address logistical challenges in classroom-based oral assessments where interactive tasks are difficult to implement. This study investigated the validity of interpretations of a prototype SDS-mediated Tourism English Speaking Test (SDS-TEST) developed using evidence-centered design. Guided by the interpretation/use argument framework, this study focused on supporting the generalization and explanation inferences. Thirty Turkish English as a Foreign Language (EFL) students in a Tourism and Hotel Management program completed three SDS tasks, each rated by four trained raters. The generalization inference was supported using a univariate generalizability study (G-study) with a fully crossed, two-facet design on 360 composite scores, while a decision study (D-study) informed optimal task and rater configurations for future designs. The explanation inference was supported through correlational analyses with related assessments and qualitative insights from participants’ strategy use gathered via stimulated recalls. Findings support the inferences in the validity argument for the SDS-TEST, highlighting the potential of SDSs in English for Specific Purposes oral assessment and offering a model for exploring validity in the next generation of oral communication assessment.
- 30 July 2026
- Published by SAGE Publications in Language Testing
- Vol. 43 (4) , 424-456
Evaluating the Validity of a Spoken Dialog System-Mediated Role-Play Test for Assessing Second Language Oral Communication
Generative AI offers new opportunities for interactive oral communication assessment for specific purposes by building on past research investigating test takers’ oral interaction with spoken dialog systems (SDSs). Performing as automated conversational agents, SDSs can address logistical challenges in classroom-based oral assessments where interactive tasks are difficult to implement. This study investigated the validity of interpretations of a prototype SDS-mediated Tourism English Speaking Test (SDS-TEST) developed using evidence-centered design. Guided by the interpretation/use argument framework, this study focused on supporting the generalization and explanation inferences. Thirty Turkish English as a Foreign Language (EFL) students in a Tourism and Hotel Management program completed three SDS tasks, each rated by four trained raters. The generalization inference was supported using a univariate generalizability study (G-study) with a fully crossed, two-facet design on 360 composite scores, while a decision study (D-study) informed optimal task and rater configurations for future designs. The explanation inference was supported through correlational analyses with related assessments and qualitative insights from participants’ strategy use gathered via stimulated recalls. Findings support the inferences in the validity argument for the SDS-TEST, highlighting the potential of SDSs in English for Specific Purposes oral assessment and offering a model for exploring validity in the next generation of oral communication assessment.
- 19 July 2026
- Published by SAGE Publications in Language Testing
- Vol. 43 (4) , 579-592
Can an AI Agent Replace Human Examiners in High-Stakes Interactive Speaking Tests? A Debate
Generative artificial intelligence (GenAI) is advancing at a remarkable speed, and its promise in transforming current practice in language assessment has been articulated by applied linguistics researchers. An emerging application of GenAI is to integrate the technology into Spoken Dialogue Systems (SDSs) to simulate human interlocutors for the purpose of speaking practice or assessment. Despite rapid technological advances, the issue of whether an AI agent can replace a human examiner in one-on-one, high-stakes interactive speaking tests remains contentious. Building on a lively academic debate on this topic at the 2025 Language Testing Research Colloquium (LTRC) in Bangkok, this Viewpoint presents arguments both for and against this proposition in terms of construct theory, practicality, washback, fairness, ethical uses of AI, and so forth.
- 19 July 2026
- Published by SAGE Publications in Language Testing
- Vol. 43 (4) , 579-592
Can an AI Agent Replace Human Examiners in High-Stakes Interactive Speaking Tests? A Debate
Generative artificial intelligence (GenAI) is advancing at a remarkable speed, and its promise in transforming current practice in language assessment has been articulated by applied linguistics researchers. An emerging application of GenAI is to integrate the technology into Spoken Dialogue Systems (SDSs) to simulate human interlocutors for the purpose of speaking practice or assessment. Despite rapid technological advances, the issue of whether an AI agent can replace a human examiner in one-on-one, high-stakes interactive speaking tests remains contentious. Building on a lively academic debate on this topic at the 2025 Language Testing Research Colloquium (LTRC) in Bangkok, this Viewpoint presents arguments both for and against this proposition in terms of construct theory, practicality, washback, fairness, ethical uses of AI, and so forth.
- 1 December 2025
- Published by EUROKD Egitm Danismanlik Group in Language Teaching Research Quarterly
- Vol. 50, 18-29
Reflections on Carol A. Chapelle’s Contributions to Applied Linguistics: Perspectives from her Students
This paper reflects on Dr Carol Ann Chapelle’s contributions to applied linguistics from the perspectives of some of her former students. As alumni of the Applied Linguistics and Technology (ALT) doctoral program at Iowa State University, we had the privilege of spending a significant amount of time with her, thus being able to closely observe her approach to research and teaching, her evolving research interests, and the increasing impact of her work in the field of applied linguistics. Our reflections on her accomplishments and contributions are centred around her academic achievements in areas such as computer-assisted language learning (CALL), computer-assisted language testing (CALT) and validity theory and validation practices; the qualities and traits that she exhibits which we believe enabled her to succeed; and her dedication to teaching, mentoring, and training of the next generation of applied linguists. Each author provided their reflections independently, based on their personal experiences of studying and collaborating with her. Dr Chapelle is a distinguished but low-key scholar. We intend to reveal the extensive efforts she has made behind the scenes to advance the knowledge and practice of language learning, teaching and assessment.
- 1 December 2025
- Published by EUROKD Egitm Danismanlik Group in Language Teaching Research Quarterly
- Vol. 50, 18-29
Reflections on Carol A. Chapelle’s Contributions to Applied Linguistics: Perspectives from her Students
This paper reflects on Dr Carol Ann Chapelle’s contributions to applied linguistics from the perspectives of some of her former students. As alumni of the Applied Linguistics and Technology (ALT) doctoral program at Iowa State University, we had the privilege of spending a significant amount of time with her, thus being able to closely observe her approach to research and teaching, her evolving research interests, and the increasing impact of her work in the field of applied linguistics. Our reflections on her accomplishments and contributions are centred around her academic achievements in areas such as computer-assisted language learning (CALL), computer-assisted language testing (CALT) and validity theory and validation practices; the qualities and traits that she exhibits which we believe enabled her to succeed; and her dedication to teaching, mentoring, and training of the next generation of applied linguists. Each author provided their reflections independently, based on their personal experiences of studying and collaborating with her. Dr Chapelle is a distinguished but low-key scholar. We intend to reveal the extensive efforts she has made behind the scenes to advance the knowledge and practice of language learning, teaching and assessment.
- Top Cited PapersOpen Access
- 24 June 2025
- Published by Wiley in TESOL Quarterly
- Vol. 59 (S1)
Exploring the Potential of Conversational AI for Assessing Second Language Oral Proficiency
Interactional Competence (IC) is an important subcomponent of oral proficiency, but many computer‐mediated oral English assessments fall short in assessing this construct mainly due to technological limitations. Spoken Dialogue Systems (SDSs) have shown promise in assessing L2 oral communication, yet further investigation is needed on their effectiveness in eliciting IC features in high‐stakes assessment contexts. This study is unique in that it analyzed both test takers' and an SDS's spoken discourse in human–computer interactions. Using an SDS to simulate an examiner in an IELTS Speaking task, the study explored how well the system mimicked human–human interaction and elicited IC features from test takers, focusing on the IC features documented in prior research. Thirty participants completed the SDS‐mediated test, with their performances rated by two trained raters. Semi‐structured interviews with 10 test takers were conducted following the assessment. The findings revealed that the SDS successfully elicited key IC features, which helped distinguish test takers at different proficiency levels, with reliable scoring across raters. Most test takers found the SDS competent, though some noted its limitations in nonverbal communication and conversational flow. These results suggest that SDSs have potential in oral proficiency assessments and provide valuable insights for refining SDS design to ensure reliable and valid assessments.
- 24 June 2025
- Published by Wiley in TESOL Quarterly
- Vol. 59 (S1)
Top Cited PapersOpen AccessExploring the Potential of Conversational AI for Assessing Second Language Oral Proficiency
Interactional Competence (IC) is an important subcomponent of oral proficiency, but many computer‐mediated oral English assessments fall short in assessing this construct mainly due to technological limitations. Spoken Dialogue Systems (SDSs) have shown promise in assessing L2 oral communication, yet further investigation is needed on their effectiveness in eliciting IC features in high‐stakes assessment contexts. This study is unique in that it analyzed both test takers' and an SDS's spoken discourse in human–computer interactions. Using an SDS to simulate an examiner in an IELTS Speaking task, the study explored how well the system mimicked human–human interaction and elicited IC features from test takers, focusing on the IC features documented in prior research. Thirty participants completed the SDS‐mediated test, with their performances rated by two trained raters. Semi‐structured interviews with 10 test takers were conducted following the assessment. The findings revealed that the SDS successfully elicited key IC features, which helped distinguish test takers at different proficiency levels, with reliable scoring across raters. Most test takers found the SDS competent, though some noted its limitations in nonverbal communication and conversational flow. These results suggest that SDSs have potential in oral proficiency assessments and provide valuable insights for refining SDS design to ensure reliable and valid assessments.
- 19 July 2023
- Published by Springer Nature
- p. 231-252
Using Spoken Dialog Systems to Assess L2 Learners’ Oral Skills in a Local Language Testing Context
- 19 July 2023
- Published by Springer Nature
- p. 231-252
Using Spoken Dialog Systems to Assess L2 Learners’ Oral Skills in a Local Language Testing Context
- Open Access
- 14 February 2022
- Published by Cambridge University Press (CUP) in ReCALL
- Vol. 34 (3) , 291-308
Exploring global online course participants’ interactions: Value of high-level engagement
Participation in online courses has become essential for training language professionals in under-resourced contexts with skills in computer-assisted language learning (CALL) (Godwin-Jones, 2014). Most online CALL courses use asynchronous computer-mediated communication (ACMC) to facilitate meaningful learning for participants. Although participants’ sustained engagement with ACMC is the target, global realities of participants interfere with their participation levels. This article investigates participants’ engagement profiles in asynchronous online discussions in an 8-week CALL-based global online course developed and implemented by a team at Iowa State University. Using a case study approach, nine focal participants’ engagement profiles have been analyzed in terms of identifying patterns of engagement in the discussion posts and their relation to the types of discussion prompts. Then, social network analysis (SNA) and thematic analysis were employed to investigate patterns of interaction among the participants in the replies. The results indicated that engagement patterns observed in discussion posts overall aligned with the primary goals of prompt types. SNA further identified two participants as social mediators to connect participants with each other. These findings are significant in that they suggest the effectiveness of using ACMC to promote co-construction of knowledge for a global audience. This article also provides implications regarding the design of discussion prompts to help maximize participant engagement with course content.
- 14 February 2022
- Published by Cambridge University Press (CUP) in ReCALL
- Vol. 34 (3) , 291-308
Open AccessExploring global online course participants’ interactions: Value of high-level engagement
Participation in online courses has become essential for training language professionals in under-resourced contexts with skills in computer-assisted language learning (CALL) (Godwin-Jones, 2014). Most online CALL courses use asynchronous computer-mediated communication (ACMC) to facilitate meaningful learning for participants. Although participants’ sustained engagement with ACMC is the target, global realities of participants interfere with their participation levels. This article investigates participants’ engagement profiles in asynchronous online discussions in an 8-week CALL-based global online course developed and implemented by a team at Iowa State University. Using a case study approach, nine focal participants’ engagement profiles have been analyzed in terms of identifying patterns of engagement in the discussion posts and their relation to the types of discussion prompts. Then, social network analysis (SNA) and thematic analysis were employed to investigate patterns of interaction among the participants in the replies. The results indicated that engagement patterns observed in discussion posts overall aligned with the primary goals of prompt types. SNA further identified two participants as social mediators to connect participants with each other. These findings are significant in that they suggest the effectiveness of using ACMC to promote co-construction of knowledge for a global audience. This article also provides implications regarding the design of discussion prompts to help maximize participant engagement with course content.
- Open Access
- 12 February 2021
- Published by National Foreign Language Resource Center in Language Learning & Technology
- Vol. 25 (1) , 44-47
Review of Automated speaking assessment: Using language technologies to score spontaneous speech
In this dynamic technology world, where innovative ideas and algorithms evolve almost daily, the landscape of computer-based language testing over the years has been dramatically reformed (Suvorov & Hegelheimer, 2013). One unique challenge that remains, however, is to effectively assess speaking proficiency (Chapelle & Voss, 2016), and since reliability plays a fundamental role in test score interpretation, the degree of error in speaking scores should be taken into account by test developers (Chapelle, 2021, p. 72). In oral speaking assessment, human rating, as traditionally the only means to evaluate constructed responses, can pose a threat to validity because of factors such as rater fatigue and rater bias (Zechner & Evanini, 2020, p. 3). A possible approach to addressing these potential threats is automated speech recognition (ASR). Recent developments such as the use of Deep Neural Network algorithms for training ASR and growth in the performance of scoring models have considerably advanced automated speech processing technology. Yet, there is still a lot to be discovered because of the lack of knowledge about how these systems work. Responding to this need, Automated Speaking Assessment: Using Language Technologies to Score Spontaneous Speech, edited by Zechner and Evanini, aims to bridge the gap between stakeholders such as scientists and engineers who may not have any expertise in reliability, validity, and fairness issues and stakeholders in academia who may not have expertise in speech processing technology. In doing so, the book unveils the state-of-the-art automated speech scoring technology, SpeechRater, used at Educational Testing Service (ETS). This book aims to be easily accessible to graduate students and applied linguists as well as testing professionals and measurement specialists.
- 12 February 2021
- Published by National Foreign Language Resource Center in Language Learning & Technology
- Vol. 25 (1) , 44-47
Open AccessReview of Automated speaking assessment: Using language technologies to score spontaneous speech
In this dynamic technology world, where innovative ideas and algorithms evolve almost daily, the landscape of computer-based language testing over the years has been dramatically reformed (Suvorov & Hegelheimer, 2013). One unique challenge that remains, however, is to effectively assess speaking proficiency (Chapelle & Voss, 2016), and since reliability plays a fundamental role in test score interpretation, the degree of error in speaking scores should be taken into account by test developers (Chapelle, 2021, p. 72). In oral speaking assessment, human rating, as traditionally the only means to evaluate constructed responses, can pose a threat to validity because of factors such as rater fatigue and rater bias (Zechner & Evanini, 2020, p. 3). A possible approach to addressing these potential threats is automated speech recognition (ASR). Recent developments such as the use of Deep Neural Network algorithms for training ASR and growth in the performance of scoring models have considerably advanced automated speech processing technology. Yet, there is still a lot to be discovered because of the lack of knowledge about how these systems work. Responding to this need, Automated Speaking Assessment: Using Language Technologies to Score Spontaneous Speech, edited by Zechner and Evanini, aims to bridge the gap between stakeholders such as scientists and engineers who may not have any expertise in reliability, validity, and fairness issues and stakeholders in academia who may not have expertise in speech processing technology. In doing so, the book unveils the state-of-the-art automated speech scoring technology, SpeechRater, used at Educational Testing Service (ETS). This book aims to be easily accessible to graduate students and applied linguists as well as testing professionals and measurement specialists.
- 1 January 2020
- Published by IGI Global Scientific Publishing
- p. 239-260
Considerations for Future Technology Development Based on EFL Teachers' Integration of Technology
Technology-enhance language learning (TELL) continues to grow in use within language classrooms. However, a number of hurdles still remain when it comes to the effective integration of technology for skill-specific language learning, such as a lack of training and an overabundance of tools to choose from. This chapter identifies and describes three major hurdles that still plague effective TELL practices. Authors describe 2 current efforts to overcome these hurdles: a Global Online Course (GOC) on effective educational technology integration, and a year-long comparative case study on the GOC that explores the trainees' perceptions of educational technology. The chapter presents potential avenues for overcoming the above hurdles based on insights gained from four teachers of the GOC, as well as the trainees' perceptions and integration of educational technology in the language classroom.
- 1 January 2020
- Published by IGI Global Scientific Publishing
- p. 239-260
Considerations for Future Technology Development Based on EFL Teachers' Integration of Technology
Technology-enhance language learning (TELL) continues to grow in use within language classrooms. However, a number of hurdles still remain when it comes to the effective integration of technology for skill-specific language learning, such as a lack of training and an overabundance of tools to choose from. This chapter identifies and describes three major hurdles that still plague effective TELL practices. Authors describe 2 current efforts to overcome these hurdles: a Global Online Course (GOC) on effective educational technology integration, and a year-long comparative case study on the GOC that explores the trainees' perceptions of educational technology. The chapter presents potential avenues for overcoming the above hurdles based on insights gained from four teachers of the GOC, as well as the trainees' perceptions and integration of educational technology in the language classroom.