GradeAid: a framework for automatic short answers grading in educational contexts—design, implementation and evaluation
Top Cited Papers
Open Access
- 19 May 2023
- journal article
- research article
- Published by Springer Nature in Knowledge and Information Systems
- Vol. 65 (10) , 4295-4334
Abstract
Automatic short answer grading (ASAG), a hot field of natural language understanding, is a research area within learning analytics. ASAG solutions are conceived to offload teachers and instructors, especially those in higher education, where classes with hundreds of students are the norm and the task of grading (short)answers to open-ended questionnaires becomes tougher. Their outcomes are precious both for the very grading and for providing students with “ad hoc” feedback. ASAG proposals have also enabled different intelligent tutoring systems. Over the years, a variety of ASAG solutions have been proposed, still there are a series of gaps in the literature that we fill in this paper. The present work proposes GradeAid, a framework for ASAG. It is based on the joint analysis of lexical and semantic features of the students’ answers through state-of-the-art regressors; differently from any other previous work, (i) it copes with non-English datasets, (ii) it has undergone a robust validation and benchmarking phase, and (iii) it has been tested on every dataset publicly available and on a new dataset (now available for researchers). GradeAid obtains performance comparable to the systems presented in the literature (root-mean-squared errors down to 0.25 based on the specific tuple $$\langle $$ dataset-question $$\rangle $$ ). We argue it represents a strong baseline for further developments in the field.
Keywords
Funding Information
- Università di Foggia
This publication has 69 references indexed in Scilit:
- Earth Mover's Distance Pooling over Siamese LSTMs for Automatic Short Answer Grading2017
- Personal Health Records: A Systematic Literature ReviewJournal of Medical Internet Research, 2017
- Hierarchical Attention Networks for Document Classification2016
- The Eras and Trends of Automatic Short Answer GradingInternational Journal of Artificial Intelligence in Education, 2015
- Interrater reliability: the kappa statisticBiochemia Medica, 2012
- Evaluating Learning Algorithms2011
- Mann‐Whitney U Test2010
- Discretion and bias in performance evaluationEuropean Economic Review, 1993
- An Analysis of Variance Test for Normality (Complete Samples)Biometrika, 1965
- Man-Computer SymbiosisIRE Transactions on Human Factors in Electronics, 1960