A novel multi-layer feature fusion-based BERT-CNN for sentence representation learning and classification
Authors:
- 2 November 2023
- journal article
- research article
- Published by Emerald Publishing in Robotic Intelligence and Automation
- Vol. 43 (6) , 704-715
Abstract
Purpose: The purpose of this paper is to describe a new approach to sentence representation learning leading to text classification using Bidirectional Encoder Representations from Transformers (BERT) embeddings. This work proposes a novel BERT-convolutional neural network (CNN)-based model for sentence representation learning and text classification. The proposed model can be used by industries that work in the area of classification of similarity scores between the texts and sentiments and opinion analysis. Design/methodology/approach: The approach developed is based on the use of the BERT model to provide distinct features from its transformer encoder layers to the CNNs to achieve multi-layer feature fusion. To achieve multi-layer feature fusion, the distinct feature vectors of the last three layers of the BERT are passed to three separate CNN layers to generate a rich feature representation that can be used for extracting the keywords in the sentences. For sentence representation learning and text classification, the proposed model is trained and tested on the Stanford Sentiment Treebank-2 (SST-2) data set for sentiment analysis and the Quora Question Pair (QQP) data set for sentence classification. To obtain benchmark results, a selective training approach has been applied with the proposed model. Findings: On the SST-2 data set, the proposed model achieved an accuracy of 92.90%, whereas, on the QQP data set, it achieved an accuracy of 91.51%. For other evaluation metrics such as precision, recall and F1 Score, the results obtained are overwhelming. The results with the proposed model are 1.17%–1.2% better as compared to the original BERT model on the SST-2 and QQP data sets. Originality/value: The novelty of the proposed model lies in the multi-layer feature fusion between the last three layers of the BERT model with CNN layers and the selective training approach based on gated pruning to achieve benchmark results.Keywords
This publication has 40 references indexed in Scilit:
- Improving task-agnostic BERT distillation with layer mapping searchNeurocomputing, 2021
- Efficient English text classification using selected Machine Learning TechniquesAlexandria Engineering Journal, 2021
- Transfer fine-tuning of BERT with phrasal paraphrasesComputer Speech & Language, 2021
- Short Text Intent Classification for Conversational Agents2020
- Attention-based sentiment analysis using convolutional and recurrent neural networkFuture Generation Computer Systems, 2020
- Research paper classification systems based on TF-IDF and LDA schemesHuman-centric Computing and Information Sciences, 2019
- Bidirectional LSTM with attention mechanism and convolutional layer for text classificationNeurocomputing, 2019
- GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understanding2018
- Sentiment Analysis Using Convolutional Neural Network2015
- Glove: Global Vectors for Word Representation2014