The Best of Two Worlds: IRT-Enhanced Automated Essay Interpretable Scoring
Open Access
- 6 April 2026
- journal article
- research article
- Published by MDPI AG in Behavioral Sciences
- Vol. 16 (4) , 542
Abstract
The Automated Essay Scoring (AES) systems confront two fundamental challenges: opaque “black-box” decision-making that limits educator trust, and insufficient validation across linguistically diverse educational contexts. This study proposes IRT-AESF, an innovative framework that bridges educational measurement theory and artificial intelligence by integrating item response theory (IRT) with deep learning. The framework generates three theoretically grounded psychometric parameters: student ability, item difficulty, and item discrimination, which provide transparent and interpretable explanations for scoring decisions. We rigorously evaluated IRT-AESF through 5-fold cross-validation on three large-scale datasets comprising 41,328 authentic essays from English and Chinese educational settings, including both classroom assessments and high-stakes examinations. Results demonstrate statistically significant improvements over competitive baseline models, achieving an 8.4% relative increase in quadratic weighted kappa while maintaining robust cross-lingual performance. This research advances the development of transparent, trustworthy automated assessment systems that deliver not only scores but meaningful diagnostic insights for educational practice.Keywords
This publication has 61 references indexed in Scilit:
- Large language models and automated essay scoring of English language learner writing: Insights into validity and reliabilityComputers and Education: Artificial Intelligence, 2024
- A Prompt-Independent and Interpretable Automated Essay Scoring Method for Chinese Second Language Writing2021
- A review of deep-neural automated essay scoring modelsBehaviormetrika, 2021
- A hierarchical classification approach to automated essay scoringAssessing Writing, 2015
- State-of-the-art automated essay scoring: Competition, results, and future directions from a United States demonstrationAssessing Writing, 2014
- Reliability and validity of rubrics for assessment through writingAssessing Writing, 2010
- QUALIFYING READERS FOR THE ONLINE SCORING NETWORK: SCORING ARGUMENT ESSAYSETS Research Report Series, 1998
- INFORMATION FUNCTIONS OF THE GENERALIZED PARTIAL CREDIT MODELETS Research Report Series, 1993
- A GENERALIZED PARTIAL CREDIT MODEL: APPLICATION OF AN EM ALGORITHMETS Research Report Series, 1992
- ESTIMATION OF LATENT ABILITY USING A RESPONSE PATTERN OF GRADED SCORES1ETS Research Bulletin Series, 1968