Deep or simple models for semantic tagging?
Authors:
Open Access
- 1 August 2020
- journal article
- research article
- Published by Association for Computing Machinery (ACM) in Proceedings of the VLDB Endowment
- Vol. 13 (12) , 2549-2562
Abstract
Semantic tagging, which has extensive applications in text mining, predicts whether a given piece of text conveys the meaning of a given semantic tag. The problem of semantic tagging is largely solved with supervised learning and today, deep learning models are widely perceived to be better for semantic tagging. However, there is no comprehensive study supporting the popular belief. Practitioners often have to train different types of models for each semantic tagging task to identify the best model. This process is both expensive and inefficient. We embark on a systematic study to investigate the following question: Are deep models the best performing model for all semantic tagging tasks? To answer this question, we compare deep models against "simple models" over datasets with varying characteristics. Specifically, we select three prevalent deep models (i.e. CNN, LSTM, and BERT) and two simple models (i.e. LR and SVM), and compare their performance on the semantic tagging task over 21 datasets. Results show that the size, the label ratio, and the label cleanliness of a dataset significantly impact the quality of semantic tagging. Simple models achieve similar tagging quality to deep models on large datasets, but the runtime of simple models is much shorter. Moreover, simple models can achieve better tagging quality than deep models when targeting datasets show worse label cleanliness and/or more severe imbalance. Based on these findings, our study can systematically guide practitioners in selecting the right learning model for their semantic tagging task.Keywords
This publication has 24 references indexed in Scilit:
- Extracting and Ranking Travel Tips from User-Generated Reviews2017
- Author-aware Aspect Topic Sentiment Model to Retrieve Supporting Opinions from Reviews2017
- XGBoost2016
- Humor Recognition and Humor Anchor Extraction2015
- Towards the Extraction of Customer-to-Customer Suggestions from Reviews2015
- Convolutional Neural Networks for Sentence Classification2014
- Answers, not links2012
- A Survey of Text Classification Algorithms2012
- Least Squares Support Vector Machine ClassifiersNeural Processing Letters, 1999
- Long Short-Term MemoryNeural Computation, 1997