Exploring the Potential of ChatGPT for Evaluating English Essays in a Criterion‐Based Assessment
Authors:
Top Cited Papers
- 3 August 2025
- journal article
- Published by Wiley in TESOL Quarterly
- Vol. 59 (S1)
Abstract
Open access to novel AI tools offers unprecedented opportunities for human–AI collaboration in writing instruction and assessment. While research on using generative AI tools like ChatGPT in these contexts is emerging, more is needed to understand their effectiveness as Automated Writing Evaluation (AWE) tools. This study explores the potential of ChatGPT (GPT‐3.5) to assist teachers and learners in the North Atlantic Treaty Organization (NATO) by evaluating English writing based on holistic scoring criteria. Using a mixed‐methods approach, the study compared ChatGPT's ratings with human ratings on 100 writing tests to assess inter‐rater reliability. It also analyzed the justifications provided by both human raters and ChatGPT to evaluate how well ChatGPT understood the rating criteria at different proficiency levels and whether its rationales could provide effective feedback for learners and support teacher feedback practices. Results showed strong agreement between ChatGPT's and human ratings, with ChatGPT demonstrating a similar understanding of the rating scales and offering justifications with elements of effective feedback. These findings indicate that ChatGPT holds promise as an AWE tool, providing meaningful feedback and valuable insights into holistic rating scales. This study encourages further exploration of AI in the L2 classroom and suggests leveraging AI to enhance writing pedagogy and classroom‐based assessment.Keywords
This publication has 38 references indexed in Scilit:
- The impact of self-revision, machine translation, and ChatGPT on L2 writing: Raters’ assessments, linguistic complexity, and error correctionAssessing Writing, 2025
- “ChatGPT is the companion, not enemies”: EFL learners’ perceptions and experiences in using ChatGPT for feedback in writingComputers and Education: Artificial Intelligence, 2024
- Comparing the quality of human and ChatGPT feedback of students’ writingLearning and Instruction, 2024
- Collaborating with ChatGPT in argumentative writing classroomsAssessing Writing, 2023
- Using ChatGPT for second language writing: Pitfalls and potentialsAssessing Writing, 2023
- Exploring the potential of using an AI language model for automated essay scoringResearch Methods in Applied Linguistics, 2023
- Exploring the Power of ChatGPT2023
- Engaging with automated writing evaluation (AWE) feedback on L2 writing: Student perceptions and revisionsAssessing Writing, 2020
- Teaching writing teachers about assessmentJournal of Second Language Writing, 2007
- Formative assessment and the design of instructional systemsInstructional Science, 1989