A multi-stage interactive writing task for the assessment of English language writing proficiency
Open Access
- 10 August 2025
- journal article
- Published by SAGE Publications in Language Testing
- Vol. 42 (4) , 423-446
Abstract
A longstanding criticism of traditional high-stakes writing assessments is their use of static prompts in which test takers compose a single text in response to a prompt. These static prompts do not allow measurement of the writing process. This paper describes the development and validation of an innovative interactive writing task. After the test taker responds to an initial prompt, a large language model automatically analyzes their response to identify the prompt themes they discussed. The test taker then receives a customized follow-up prompt asking them to discuss one of the themes they did not address in their original response. Several experimental results support the task’s validity. Results showed that prompt phrasing is important to elicit responses most consistent with the task expectations and that test takers are able to produce responses that are relevant. Regarding issues of fairness in administration, we found that scores are not negatively affected by the type of follow-up task. We found that an accurate theme-detection model ensures test takers are not asked to write about something they addressed in their initial response, which could threaten test score validity claims. This design brings high-stakes writing tasks into closer alignment with writing models that frame writing as a process.Keywords
This publication has 35 references indexed in Scilit:
- Linguistic, cultural and substantive patterns in L2 writing: A qualitative illustration of MisLevy’s sociocognitive perspective on assessmentAssessing Writing, 2022
- Remodeling writers’ composing processes: Implications for writing assessmentAssessing Writing, 2021
- Second language writing instructors’ feedback practice in response to automated writing evaluation: A sociocultural perspectiveSystem, 2020
- Source use in the story continuation writing taskAssessing Writing, 2019
- A Revised Writer(s)-Within-Community Model of WritingEducational Psychologist, 2018
- Second language writing as sociocognitive alignmentJournal of Second Language Writing, 2015
- Using multiple texts in an integrated writing assessment: Source text use as a predictor of scoreJournal of Second Language Writing, 2013
- The Process Writing Approach: A Meta-analysisThe Journal of Educational Research, 2011
- Language Testing and Validation2005
- Information Retrieval, 2nd ed. C.J. Van Rijsbergen. London: Butterworths; 1979: 208 pp. Price: $32.50Journal of the American Society for Information Science, 1979