ORDisCo: Effective and Efficient Usage of Incremental Unlabeled Data for Semi-supervised Continual Learning
Authors:
- 1 June 2021
- conference paper
- Published by Institute of Electrical and Electronics Engineers (IEEE)
- p. 5379-5388
Abstract
Continual learning usually assumes the incoming data are fully labeled, which might not be applicable in real applications. In this work, we consider semi-supervised continual learning (SSCL) that incrementally learns from partially labeled data. Observing that existing continual learning methods lack the ability to continually exploit the unlabeled data, we propose deep Online Replay with Discriminator Consistency (ORDisCo) to interdependently learn a classifier with a conditional generative adversarial network (GAN), which continually passes the learned data distribution to the classifier. In particular, ORDisCo replays data sampled from the conditional generator to the classifier in an online manner, exploiting unlabeled data in a time- and storage-efficient way. Further, to explicitly overcome the catastrophic forgetting of unlabeled data, we selectively stabilize parameters of the discriminator that are important for discriminating the pairs of old unlabeled data and their pseudo-labels predicted by the classifier. We extensively evaluate ORDisCo on various semi-supervised learning benchmark datasets for SSCL, and show that ORDisCo achieves significant performance improvement on SVHN, CIFAR10 and Tiny-ImageNet, compared to strong baselines.Keywords
All Related Versions
Funding Information
- Research and Development
This publication has 19 references indexed in Scilit:
- Semi-supervised stacked autoencoder-based deep hierarchical semantic feature for real-time fingerprint liveness detectionJournal of Real-Time Image Processing, 2019
- Continual lifelong learning with neural networks: A reviewNeural Networks, 2019
- Incremental semi-supervised learning on streaming dataPattern Recognition, 2019
- Virtual Adversarial Training: A Regularization Method for Supervised and Semi-Supervised LearningIEEE Transactions on Pattern Analysis and Machine Intelligence, 2018
- Smooth Neighbors on Teacher Graphs for Semi-Supervised Learning2018
- Overcoming catastrophic forgetting in neural networksProceedings of the National Academy of Sciences, 2017
- An Adaptive Semisupervised Feature Analysis for Video Semantic RecognitionIEEE Transactions on Cybernetics, 2017
- Deformed Graph Laplacian for Semisupervised LearningIEEE Transactions on Neural Networks and Learning Systems, 2015
- Introduction to Semi-Supervised LearningSynthesis Lectures on Artificial Intelligence and Machine Learning, 2009
- Semi-supervised PCA-Based Face Recognition Using Self-training2006