Short text automatic scoring system based on BERT-BiLSTM model
Authors:
- Linzhong XiaLXLinzhong Xia
- Jianfeng YeJYJianfeng Ye
- De’an LuoDLDe’an Luo
- Mingxiang GuanMGMingxiang Guan
- Jun LiuJLJun Liu
- Xuemei CaoXCXuemei Cao
- 1 May 2022
- journal article
- Published by China Science Publishing & Media Ltd. in JOURNAL OF SHENZHEN UNIVERSITY SCIENCE AND ENGINEERING
- Vol. 39 (3) , 349-354
Abstract
Aiming at the problems of sparse features, polysemy of one word and less context related information in short text automatic scoring, a short text automatic scoring model based on bidirectional encoder representations from transformers - bidirectional long short-term memory (BERT-BiLSTM) is proposed. Firstly, the large-scale corpus is pre-trained with bidirectional encoder representations from transformers (BERT) language model to acquire the semantic features of the general language. Then the semantic features of short text and the semantics of keywords in a specific context are acquired through the short text data for the pre-fine tuning downstream specific tasks set pre-fined by BERT. And then the deep-seated context dependency is captured through bidirectional long short-term memory (BiLSTM). Finally, the obtained feature vectors are input into Softmax regression model for automatic scoring. The experimental results show that compared with other benchmark models of convolutional neural networks(CNN), character-level CNN (CharCNN), long short-term memory (LSTM) and BERT, the short text automatic scoring model based on BERT-BiLSTM achieves the best average value of quadratic weighted kappa coefficient.Keywords
This publication has 11 references indexed in Scilit:
- An LSTM-Based Method with Attention Mechanism for Travel Time PredictionSensors, 2019
- Describing Video With Attention-Based Bidirectional LSTMIEEE Transactions on Cybernetics, 2018
- A discourse-aware neural network-based text model for document-level text classificationJournal of Information Science, 2017
- Sentiment-Specific Representation Learning for Document-Level Sentiment Analysis2015
- Glove: Global Vectors for Word Representation2014
- Framewise phoneme classification with bidirectional LSTM and other neural network architecturesNeural Networks, 2005
- Seeing stars2005
- C-rater: Automated Scoring of Short-Answer QuestionsComputers and the Humanities, 2003
- Long Short-Term MemoryNeural Computation, 1997
- Indexing by latent semantic analysisJournal of the American Society for Information Science, 1990