Evaluating the Validity of a Spoken Dialog System-Mediated Role-Play Test for Assessing Second Language Oral Communication
- 30 July 2026
- journal article
- research article
- Published by SAGE Publications in Language Testing
- Vol. 43 (4) , 424-456
Abstract
Generative AI offers new opportunities for interactive oral communication assessment for specific purposes by building on past research investigating test takers’ oral interaction with spoken dialog systems (SDSs). Performing as automated conversational agents, SDSs can address logistical challenges in classroom-based oral assessments where interactive tasks are difficult to implement. This study investigated the validity of interpretations of a prototype SDS-mediated Tourism English Speaking Test (SDS-TEST) developed using evidence-centered design. Guided by the interpretation/use argument framework, this study focused on supporting the generalization and explanation inferences. Thirty Turkish English as a Foreign Language (EFL) students in a Tourism and Hotel Management program completed three SDS tasks, each rated by four trained raters. The generalization inference was supported using a univariate generalizability study (G-study) with a fully crossed, two-facet design on 360 composite scores, while a decision study (D-study) informed optimal task and rater configurations for future designs. The explanation inference was supported through correlational analyses with related assessments and qualitative insights from participants’ strategy use gathered via stimulated recalls. Findings support the inferences in the validity argument for the SDS-TEST, highlighting the potential of SDSs in English for Specific Purposes oral assessment and offering a model for exploring validity in the next generation of oral communication assessment.Keywords
Funding Information
- Iowa State University (Iowa State University English Department Graduate)
This publication has 57 references indexed in Scilit:
- Human‐ versus artificial intelligence‐delivered roleplay tasks for assessing interactional competence: An applied conversation analytic studyTESOL Quarterly, 2025
- Intent‐Based Versus GPT ‐Based Conversational Agents: Benefits and Challenges for Practicing and Assessing Oral InteractionTESOL Quarterly, 2025
- Exploring the Potential of Conversational AI for Assessing Second Language Oral ProficiencyTESOL Quarterly, 2025
- A systematic review on artificial intelligence dialogue systems for enhancing English as foreign language students’ interactional competence in the universityComputers and Education: Artificial Intelligence, 2023
- “Doing Explicit” in hospitality and tourism service encounters in English as a lingua francaEnglish for Specific Purposes, 2023
- The Development and Evaluation of Interactional Competence Elicitor for Oral Language AssessmentsETS Research Report Series, 2021
- Communicative language teaching and English as a foreign language undergraduates’ communicative competence in Tourism EnglishJournal of Hospitality, Leisure, Sport & Tourism Education, 2020
- Raising explicitness through self-repair in English as a lingua francaJournal of Pragmatics, 2011
- THE SPEAKING SECTION OF THE TOEFL IBT™ (SSTIBT): TEST‐TAKERS' REPORTED STRATEGIC BEHAVIORSETS Research Report Series, 2009
- The service encounter2008