Variational Data-Free Knowledge Distillation for Continual Learning
- 1 May 2023
- journal article
- research article
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Transactions on Pattern Analysis and Machine Intelligence
- Vol. 45 (10) , 12618-12634
Abstract
Deep neural networks suffer from catastrophic forgetting when trained on sequential tasks in continual learning. Various methods rely on storing data of previous tasks to mitigate catastrophic forgetting, which is prohibited in real-world applications considering privacy and security issues. In this paper, we consider a realistic setting of continual learning, where training data of previous tasks are unavailable and memory resources are limited. We contribute a novel knowledge distillation-based method in an information-theoretic framework by maximizing mutual information between outputs of previously learned and current networks. Due to the intractability of computation of mutual information, we instead maximize its variational lower bound, where the covariance of variational distribution is modeled by a graph convolutional network. The inaccessibility of data of previous tasks is tackled by Taylor expansion, yielding a novel regularizer in network training loss for continual learning. The regularizer relies on compressed gradients of network parameters. It avoids storing previous task data and previously learned networks. Additionally, we employ self-supervised learning technique for learning effective features, which improves the performance of continual learning. We conduct extensive experiments including image classification and semantic segmentation, and the results show that our method achieves state-of-the-art performance on continual learning benchmarks.Keywords
Funding Information
- National Key R&D Program of China (2021YFA1003002)
- National Natural Science Foundation of China (12125104, U20B2075, 12090021, 11971373, 61721002)
- Fundamental Research Funds for the Central Universities
This publication has 56 references indexed in Scilit:
- DeepLab: Semantic Image Segmentation with Deep Convolutional Nets, Atrous Convolution, and Fully Connected CRFsIEEE Transactions on Pattern Analysis and Machine Intelligence, 2017
- Overcoming catastrophic forgetting in neural networksProceedings of the National Academy of Sciences of the United States of America, 2017
- Unsupervised Learning of Visual Representations by Solving Jigsaw Puzzles2016
- Identity Mappings in Deep Residual Networks2016
- The Cityscapes Dataset for Semantic Urban Scene Understanding2016
- Mastering the game of Go with deep neural networks and tree searchNature, 2016
- The Algorithmic Foundations of Differential PrivacyFoundations and Trends® in Theoretical Computer Science, 2014
- Training feed-forward networks with the extended Kalman algorithm2003
- Gradient-based learning applied to document recognitionProceedings of the IEEE, 1998
- Catastrophic Interference in Connectionist Networks: The Sequential Learning Problem1989