Training Networks in Null Space of Feature Covariance for Continual Learning
Authors:
- 1 June 2021
- conference paper
- Published by Institute of Electrical and Electronics Engineers (IEEE)
- p. 184-193
Abstract
In the setting of continual learning, a network is trained on a sequence of tasks, and suffers from catastrophic forgetting. To balance plasticity and stability of network in continual learning, in this paper, we propose a novel network training algorithm Adam-NSCL which sequentially optimizes network parameters in the null space of all previous tasks. We first propose two mathematical conditions respectively for achieving network stability and plasticity in continual learning. Based on them, the network training for sequential tasks without forgetting can be simply achieved by projecting the candidate parameter update into the approximate null space of all previous tasks in the network training process, where the candidate parameter update can be generated by Adam. The approximate null space can be derived by applying singular value decomposition to the un-centered covariance matrix of all input features of previous tasks for each linear layer. For efficiency, the uncentered covariance matrix can be incrementally computed after learning each task. We also empirically verify the rationality of the approximate null space at each linear layer. We apply our approach to training networks for continual learning on benchmark datasets of CIFAR-100 and TinyImageNet, and the results suggest that the proposed approach outperforms or matches the state-ot-the-art continual learning approaches.Keywords
All Related Versions
This publication has 23 references indexed in Scilit:
- Deep Learning Human Mind for Automated Visual Classification2017
- iCaRL: Incremental Classifier and Representation Learning2017
- Overcoming catastrophic forgetting in neural networksProceedings of the National Academy of Sciences of the United States of America, 2017
- Principal component analysis: a review and recent developmentsPhilosophical Transactions of the Royal Society A: Mathematical, Physical and Engineering Sciences, 2016
- Mastering the game of Go with deep neural networks and tree searchNature, 2016
- ImageNet Large Scale Visual Recognition ChallengeInternational Journal of Computer Vision, 2015
- The stability-plasticity dilemma: investigating the continuum from catastrophic forgetting to age-limited learning effectsFrontiers in Psychology, 2013
- Catastrophic forgetting in connectionist networksTrends in Cognitive Sciences, 1999
- Connectionist models of recognition memory: Constraints imposed by learning and forgetting functions.Psychological Review, 1990
- Catastrophic Interference in Connectionist Networks: The Sequential Learning Problem1989