Abstract
Traditional Automated Essay Scoring (AES) methods typically require large amounts of labeled data and extensive model training. Recent advances in Large Language Models (LLMs) have enabled zero-shot AES methods that bypass these requirements. In this paper, we propose Masked Cloze Scoring (MCS), a novel zero-shot method that leverages LLMs’ masked language modeling capacity as its core mechanism for evaluating how well an essay aligns with model predictions. Unlike prior methods that prompt LLMs to assign scores directly, MCS integrates two complementary signals: in-context word prediction for micro-level lexical appropriateness, and relative essay length as a proxy for macro-level writing quality. Experimental results demonstrate that MCS consistently outperforms existing training-free zero-shot baselines on two widely used benchmark datasets, ASAP and TOEFL11. Compared with LCES, the state-of-the-art zero-shot method that requires additional training, MCS underperforms on TOEFL11 but achieves comparable performance on ASAP. Moreover, MCS attains performance comparable to that of three strong supervised cross-prompt baselines on ASAP. Beyond holistic scoring, MCS produces word-level diagnostic feedback, offering added pedagogical benefits. Overall, MCS advances the state of training-free zero-shot AES, delivering competitive performance without requiring labeled data or additional training.

This publication has 10 references indexed in Scilit: