A Chinese Few-Shot Text Classification Method Utilizing Improved Prompt Learning and Unlabeled Data
Open Access
- 6 March 2023
- journal article
- research article
- Published by MDPI AG in Applied Sciences
- Vol. 13 (5) , 3334
Abstract
Insufficiently labeled samples and low-generalization performance have become significant natural language processing problems, drawing significant concern for few-shot text classification (FSTC). Advances in prompt learning have significantly improved the performance of FSTC. However, prompt learning methods typically require the pre-trained language model and tokens of the vocabulary list for model training, while different language models have different token coding structures, making it impractical to build effective Chinese prompt learning methods from previous approaches related to English. In addition, a majority of current prompt learning methods do not make use of existing unlabeled data, thus often leading to unsatisfactory performance in real-world applications. To address the above limitations, we propose a novel Chinese FSTC method called CIPLUD that combines an improved prompt learning method and existing unlabeled data, which are used for the classification of a small amount of Chinese text data. We used the Chinese pre-trained language model to build two modules: the Multiple Masks Optimization-based Prompt Learning (MMOPL) module and the One-Class Support Vector Machine-based Unlabeled Data Leveraging (OCSVM-UDL) module. The former generates prompt prefixes with multiple masks and constructs suitable prompt templates for Chinese labels. It optimizes the random token combination problem during label prediction with joint probability and length constraints. The latter, by establishing an OCSVM model in the trained text vector space, selects reasonable pseudo-label data for each category from a large amount of unlabeled data. After selecting the pseudo-label data, we mixed them with the previous few-shot annotated data to obtain brand new training data and then repeated the steps of the two modules as an iterative semi-supervised optimization process. The experimental results on the four Chinese FSTC benchmark datasets demonstrate that our proposed solution outperformed other prompt learning methods with an average accuracy improvement of 2.3%.Keywords
Funding Information
- National Social Science Foundation Western Project of China (19XTQ010)
This publication has 28 references indexed in Scilit:
- How Can We Know What Language Models Know?Transactions of the Association for Computational Linguistics, 2020
- Learning transferable features in meta-learning for few-shot text classificationPattern Recognition Letters, 2020
- Few-Shot Transfer Learning for Text Classification With Lightweight Word Embedding Based ModelsIEEE Access, 2019
- Survey on supervised machine learning techniques for automatic text classificationArtificial Intelligence Review, 2019
- Temporal specificity-based text classification for information retrievalTurkish Journal of Electrical Engineering and Computer Sciences, 2018
- Learning to Compare: Relation Network for Few-Shot Learning2018
- Few-shot learning for short text classificationMultimedia Tools and Applications, 2018
- A comparative evaluation of outlier detection algorithms: Experiments and analysesPattern Recognition, 2018
- Breaking the Closed World Assumption in Text Classification2016
- Estimating the Support of a High-Dimensional DistributionNeural Computation, 2001