Looking Back on the Past: Active Learning With Historical Evaluation Results
- 18 December 2020
- journal article
- research article
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Transactions on Knowledge and Data Engineering
- Vol. 34 (10) , 4921-4932
Abstract
Active learning is an effective approach for tasks with limited labeled data. It samples a small set of data to annotate actively and is widely applied in various AI tasks. It uses an iterative process, during which we utilize the current trained model to evaluate all unlabeled samples and annotate the best samples based on a specific query strategy to update the underlying model iteratively. Most existing active learning approaches rely on only the evaluation results generated by the current model and ignore the results from previous iterations. In this paper, we propose using more historical evaluation results which can provide additional information to help better select samples. First, we apply two kinds of heuristic features of the historical evaluation results, the weighted sum of and fluctuation of the historical evaluation sequence, to improve the effectiveness of sampling. Next, to use the information contained in the historical results more globally, we design a novel query strategy that learns how to select samples based on the historical sequences automatically. We also improve current state-of-the-art active learning methods by introducing historical evaluation results. Experimental results on two common NLP tasks including text classification and named entity recognition show that our methods significantly outperform current methodsKeywords
Funding Information
- National Natural Science Foundation of China (No. 61872370, No. 61832017)
- Natural Science Foundation of Shandong Province (ZR2019ZD06)
- Beijing Outstanding Young Scientist Program (NO. BJJWZYJH012019100020098)
This publication has 38 references indexed in Scilit:
- Active Learning for Ranking through Expected Loss OptimizationIEEE Transactions on Knowledge and Data Engineering, 2014
- Active Learning by Querying Informative and Representative ExamplesIEEE Transactions on Pattern Analysis and Machine Intelligence, 2014
- Uncertainty-based active learning with instability estimation for text classificationACM Transactions on Speech and Language Processing, 2012
- A hybrid neural network and ARIMA model for water quality time series predictionEngineering Applications of Artificial Intelligence, 2010
- Adapting boosting for information retrieval measuresInformation Retrieval Journal, 2009
- Active Learning with History-Based Query Selection for Text Categorisation2007
- Applying LSTM to Time Series Predictable through Time-Window Approaches2001
- Reinforcement Learning: An IntroductionIEEE Transactions on Neural Networks, 1998
- A modified Mann-Kendall trend test for autocorrelated dataJournal of Hydrology, 1998
- On Information and SufficiencyThe Annals of Mathematical Statistics, 1951