Snowball: Iterative Model Evolution and Confident Sample Discovery for Semi-Supervised Learning on Very Small Labeled Datasets
- 25 May 2020
- journal article
- research article
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Transactions on Multimedia
- Vol. 23 (15209210) , 1354-1366
Abstract
In this work, we develop a joint sample discovery and iterative model evolution method for semi-supervised learning on very small labeled training sets. We propose a master-teacher-student model framework to provide multi-layer guidance during the model evolution process with multiple iterations and generations. The teacher model is constructed by performing an exponential moving average of the student models obtained from past training steps. The master network combines the knowledge of the student and teacher models with additional access to newly discovered samples. The master and teacher models are then used to guide the training of the student network by enforcing the consistency between their predictions of unlabeled samples and evolve all models when more and more samples are discovered. Our extensive experiments demonstrate that the process of discovering confident samples from the unlabeled dataset, once coupled with the master-teacher-student network evolution, can significantly improve the overall semi-supervised learning performance. For example, on the CIFAR-10 dataset, with a small set of 250 labeled samples, our method achieves an error rate of 11.58%, more than 38% lower than Mean-Teacher (49.91%). When coupled with the MixMatch augmentation and loss function, the improvements are also significant.Keywords
Funding Information
- National Science Foundation (1647213, 1646065)
This publication has 26 references indexed in Scilit:
- Semi-Supervised Bi-Dictionary Learning for Image Classification With Smooth Representation-Based Label PropagationIEEE Transactions on Multimedia, 2016
- Self-labeled techniques for semi-supervised learning: taxonomy, software and empirical studyKnowledge and Information Systems, 2013
- Semi-Supervised Learning (Chapelle, O. et al., Eds.; 2006) [Book reviews]IEEE Transactions on Neural Networks, 2009
- Semi-supervised learning using randomized mincuts2004
- Consumer perceived risk: conceptualisations and modelsEuropean Journal of Marketing, 1999
- Combining labeled and unlabeled data with co-training1998
- Bagging predictorsMachine Learning, 1996
- Unsupervised word sense disambiguation rivaling supervised methods1995
- Training with Noise is Equivalent to Tikhonov RegularizationNeural Computation, 1995
- Creating artificial neural networks that generalizeNeural Networks, 1991