Reducing English Major Students’ Writing Errors With an Automated Writing Evaluation System: Evidence From Eye-Tracking Technology
Open Access
- 5 March 2025
- journal article
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Transactions on Learning Technologies
- Vol. 18, 304-320
Abstract
Much research has applied automated writing evaluation (AWE) systems to English writing instruction; however, understanding how students internalize and apply this feedback to reduce writing errors is difficult, largely due to the personal and private nature of this process. Therefore, this research utilized eye-tracking technology to explore the AWE system's effectiveness in reducing the writing errors of English major students. A total of 118 higher vocational college students majoring in English in China participated in this eight-week study. The experimental group studied with and received feedback from both the AWE system (Pigai) and the teacher, whereas the control group studied without the AWE system and only received teacher feedback. Eye-tracking experiments were conducted before and after the writing instruction. Participants' responses during the eye-tracking experiment, first-person eye movement video data, and corresponding gaze data were collected. Leveraging the application of neural network technology in optical character recognition (OCR), combined with data from an eye-tracking device, we developed a system that can transform first-person eye movement video data and gaze data into heatmaps and eye-tracking indices conducive to analysis. Various data analysis methods were employed, including neural network algorithms, heatmap analysis, Mann-Whitney U test, independent-samples t-test, and Welch's t-test. The results for the post-eye-tracking experiment responses, heatmaps, and eye-tracking indices indicate the advantages of using the AWE system, which effectively enhances students' ability to recognize writing errors while reducing processing time, by facilitating the internalization of writing errors through continuous feedback on such errors, and enabling them to apply this knowledge to new materials, thereby recognizing writing errors more quickly and accurately, and thus helping them to reduce writing errors. The pedagogical implications are fully discussed.Keywords
Funding Information
- Research Scheme from the University Research Facility (04884)
- Faculty of Liberal Arts and Social Sciences, Education University of Hong Kong
- Dean's Research Fund
- Education University of Hong Kong
This publication has 68 references indexed in Scilit:
- Automated writing evaluation systems: A systematic review of Grammarly, Pigai, and Criterion with a perspective on future directions in the age of generative artificial intelligenceEducation and Information Technologies, 2024
- Learner interaction with, and response to, AI-programmed automated writing evaluation feedback in EFL writing: An exploratory studyEducation and Information Technologies, 2023
- Student engagement with automated written corrective feedback (AWCF) provided by Grammarly: A multiple case studyAssessing Writing, 2020
- Engaging with automated writing evaluation (AWE) feedback on L2 writing: Student perceptions and revisionsAssessing Writing, 2020
- Scenes, Saliency Maps and Scanpaths2019
- Teacher implementation of Self-Regulated Strategy Development with an automated writing evaluation system: Effects on the argumentative writing performance of middle school studentsContemporary Educational Psychology, 2018
- Automated Writing Assessment2018
- Automated essay evaluation software in English Language Arts classrooms: Effects on teacher feedback, student motivation, and writing qualityComputers & Education, 2016
- A hierarchical classification approach to automated essay scoringAssessing Writing, 2015
- Eye movements in reading words and sentences2007