An Empirical Analysis of BERT Embedding for Automated Essay Scoring
Authors:
- Majdi BeseisoMBMajdi Beseiso
- Saleh AlzahraniSASaleh Alzahrani
Open Access
- 1 January 2020
- journal article
- research article
- Published by The Science and Information Organization in International Journal of Advanced Computer Science and Applications
- Vol. 11 (10)
Abstract
Automated Essay Scoring (AES) is one of the most challenging problems in Natural Language Processing (NLP). The significant challenges include the length of the essay, the presence of spelling mistakes affecting the quality of the essay and representing essay in terms of relevant features for the efficient scoring of essays. In this work, we present a comparative empirical analysis of Automatic Essay Scoring (AES) models based on combinations of various feature sets. We use 30-manually extracted features, 300-word2vec representation, and 768-word embedding features using BERT model and forms different combinations for evaluating the performance of AES models. We formulate an automated essay scoring problem as a rescaled regression problem and quantized classification problem. We analyzed the performance of AES models for different combinations. We compared them against the existing ensemble approaches in terms of Kappa Statistics and Accuracy for rescaled regression problem and quantized classification problem respectively. A combination of 30-manually extracted features, 300-word2vec representation, and 768-word embedding features using BERT model results up to 77.2 ± 1.7 of Kappa statistics for rescaled regression problem and 75.2 ± 1.0 of accuracy value for Quantized Classification problem using a benchmark dataset consisting of about 12,000 essays divided into eight groups. The reporting results provide directions to the researchers in the field to use manually extracted features along with deep encoded features for developing a more reliable AES model.Keywords
This publication has 7 references indexed in Scilit:
- Automated Essay Scoring: A Survey of the State of the Art2019
- Argument Mining for Improving the Automated Scoring of Persuasive EssaysProceedings of the AAAI Conference on Artificial Intelligence, 2018
- Automated Assessment of Non-Native Learner Essays: Investigating the Role of Linguistic FeaturesInternational Journal of Artificial Intelligence in Education, 2017
- Automatic Text Scoring Using Neural Networks2016
- Automated Essay Scoring by Capturing Relative Writing QualityThe Computer Journal, 2013
- An Unsupervised Automated Essay Scoring SystemIEEE Intelligent Systems, 2010
- Evaluation of text coherence for electronic essay scoring systemsNatural Language Engineering, 2004