Evaluating Unsupervised Text Classification: Zero-shot and Similarity-based Approaches
Open Access
- 16 December 2022
- conference paper
- Published by Association for Computing Machinery (ACM)
- p. 6-15
Abstract
Text classification of unseen classes is a challenging Natural Language Processing task and is mainly attempted using two different types of approaches. Similarity-based approaches attempt to classify instances based on similarities between text document representations and class description representations. Zero-shot text classification approaches aim to generalize knowledge gained from a training task by assigning appropriate labels of unknown classes to text documents. Although existing studies have already investigated individual approaches to these categories, the experiments in literature do not provide a consistent comparison. This paper addresses this gap by conducting a systematic evaluation of different similarity-based and zero-shot approaches for text classification of unseen classes. Different state-of-the-art approaches are benchmarked on four text classification datasets, including a new dataset from the medical domain. Additionally, novel SimCSE [7] and SBERT-based [26] baselines are proposed, as other baselines used in existing work yield weak classification results and are easily outperformed. Finally, the novel similarity-based Lbl2TransformerVec approach is presented, which outperforms previous state-of-the-art approaches in unsupervised text classification. Our experiments show that similarity-based approaches significantly outperform zero-shot approaches in most cases. Additionally, using SimCSE or SBERT embeddings instead of simpler text representations increases similarity-based classification results even further.Keywords
All Related Versions
This publication has 30 references indexed in Scilit:
- Text Classification Algorithms: A SurveyInformation, 2019
- Survey on supervised machine learning techniques for automatic text classificationArtificial Intelligence Review, 2019
- A Survey of Zero-Shot LearningACM Transactions on Intelligent Systems and Technology, 2019
- Integrating Semantic Knowledge to Tackle Zero-shot Text Classification2019
- A recent overview of the state-of-the-art elements of text classificationExpert Systems with Applications, 2018
- Few-Shot and Zero-Shot Multi-Label Learning for Structured Label Spaces2018
- A Broad-Coverage Challenge Corpus for Sentence Understanding through Inference2018
- FEVER: a Large-scale Dataset for Fact Extraction and VERification2018
- NewsWeeder: Learning to Filter Netnews1995
- Indexing by latent semantic analysisJournal of the American Society for Information Science, 1990