Toward Standardized Interactivity: An AI-Enabled Adaptive Speaking Task for Computer-Delivered Large-Scale Assessment
Open Access
- 27 July 2026
- journal article
- research article
- Published by SAGE Publications in Language Testing
- Vol. 43 (4) , 457-480
Abstract
Despite task interactivity eliciting different aspects of language use, more interactive, adaptive speaking tasks have been challenging to implement in computer-delivered large-scale high-stakes contexts. Recent advances in large language models (LLMs) offer a potential means to address this tension through spoken dialogue systems (SDSs) but fall short of full adaptivity. To enhance adaptivity in an LLM-enhanced SDS-delivered interview task, this article proposes two solutions: (1) LLM-driven response evaluation using question-specific task completion rubrics for adaptive follow-up question selection and (2) item response theory (IRT) scoring accounting for prompt variability. We examined the extent to which these solutions functioned to simulate adaptivity, from 5,909 test takers’ responses to an adaptive speaking task with an avatar and a non-adaptive monologic task. Automated task completion evaluations were comparable to expert ratings, though follow-up question evaluation showed low agreement both among human raters and between raters and the system. IRT scoring yielded higher test–retest reliability, and the adaptive task elicited more reciprocal language use than the monologic task. Unlike earlier SDSs that prioritized consistency over adaptivity, the real-time meaning-level evaluation with IRT modeling strikes a balance between standardization and adaptivity. These findings support the viability of adaptive speaking tasks in computer-based standardized assessments.Keywords
This publication has 52 references indexed in Scilit:
- Human‐ versus artificial intelligence‐delivered roleplay tasks for assessing interactional competence: An applied conversation analytic studyTESOL Quarterly, 2025
- Gnowsis: Multimodal multitask learning for oral proficiency assessmentsComputer Speech & Language, 2025
- Exploring the Potential of Conversational AI for Assessing Second Language Oral ProficiencyTESOL Quarterly, 2025
- Can AI provide useful holistic essay scoring?Computers and Education: Artificial Intelligence, 2024
- Large language models and automated essay scoring of English language learner writing: Insights into validity and reliabilityComputers and Education: Artificial Intelligence, 2024
- Using Multi-faceted Rasch Analysis to Examine Raters, Prompts, and Rubrics in an Office-Hour Role-Play Task2023
- Best Practices for Constructed‐Response ScoringETS Research Report Series, 2022
- Identifying Linguistic Markers of Collaboration in Second Language Peer Interaction: A Lexico‐grammatical ApproachTESOL Quarterly, 2018
- Implementing a Contributory Scoring Approach for the GRE® Analytical Writing Section: A Comprehensive Empirical InvestigationETS Research Report Series, 2017
- Academic talk in American university classrooms: crossing the boundaries of oral‐literate discourse?Journal of English for Academic Purposes, 2006