Masked Cloze Scoring (MCS): Zero-Shot Automated Essay Scoring via Masked Predictions from Large Language Models
Authors:
Preprint
- 14 February 2026
- preprint
- Published by Elsevier in SSRN Electronic Journal
Abstract
Traditional Automated Essay Scoring (AES) methods typically require large amounts of labeled data and extensive model training. Recent advances in Large Language Models (LLMs) have enabled zero-shot AES methods that bypass these requirements. In this paper, we propose Masked Cloze Scoring (MCS), a novel zero-shot method that leverages LLMs’ masked language modeling capacity as its core mechanism for evaluating how well an essay aligns with model predictions. Unlike prior methods that prompt LLMs to assign scores directly, MCS integrates two complementary signals: in-context word prediction for micro-level lexical appropriateness, and relative essay length as a proxy for macro-level writing quality. Experimental results demonstrate that MCS consistently outperforms existing training-free zero-shot baselines on two widely used benchmark datasets, ASAP and TOEFL11. Compared with LCES, the state-of-the-art zero-shot method that requires additional training, MCS underperforms on TOEFL11 but achieves comparable performance on ASAP. Moreover, MCS attains performance comparable to that of three strong supervised cross-prompt baselines on ASAP. Beyond holistic scoring, MCS produces word-level diagnostic feedback, offering added pedagogical benefits. Overall, MCS advances the state of training-free zero-shot AES, delivering competitive performance without requiring labeled data or additional training.Keywords
This publication has 10 references indexed in Scilit:
- Exploring the difference and quality of AI-generated versus human-written textsDiscover Education, 2025
- Conundrums in Cross-Prompt Automated Essay Scoring: Making Sense of the State of the Art2024
- A large-scale comparison of human-written versus ChatGPT-generated essaysScientific Reports, 2023
- Exploring the potential of using an AI language model for automated essay scoringResearch Methods in Applied Linguistics, 2023
- PMAES: Prompt-mapping Contrastive Learning for Cross-prompt Automated Essay Scoring2023
- Automated Essay Scoring with Discourse-Aware Neural Models2019
- Automatic Text Scoring Using Neural Networks2016
- Evaluation of text coherence for electronic essay scoring systemsNatural Language Engineering, 2004
- The imminence of grading essays by computer—25 years laterComputers and Composition, 1993
- A Coefficient of Agreement for Nominal ScalesEducational and Psychological Measurement, 1960