Incremental Concept Learning via Online Generative Memory Recall
- 6 August 2020
- journal article
- research article
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Transactions on Neural Networks and Learning Systems
- Vol. 32 (7) , 3206-3216
Abstract
The ability to learn more concepts from incrementally arriving data over time is essential for the development of a lifelong learning system. However, deep neural networks often suffer from forgetting previously learned concepts when continually learning new concepts, which is known as the catastrophic forgetting problem. The main reason for catastrophic forgetting is that past concept data are not available, and neural weights are changed during incrementally learning new concepts. In this article, we propose an incremental concept learning framework that includes two components, namely, ICLNet and RecallNet. ICLNet, which consists of a trainable feature extractor and a dynamic concept memory matrix, aims to learn new concepts incrementally. We propose a concept-contrastive loss to alleviate the magnitude of neural weight changes and mitigate the catastrophic forgetting problems. RecallNet aims to consolidate old concepts memory and recall pseudo samples, whereas ICLNet learns new concepts. We propose a balanced online memory recall strategy to reduce the information loss of old concept memory. We evaluate the proposed approach on the MNIST, Fashion-MNIST, and SVHN data sets and compare it with other pseudorehearsal-based approaches. Extensive experiments demonstrate the effectiveness of our approach.Keywords
All Related Versions
Funding Information
- National Key Research and Development Program of China (2018AAA0101005)
- National Natural Science Foundation of China (61832016, 61672520, 61720106006)
This publication has 21 references indexed in Scilit:
- Neural plasticity across the lifespanWIREs Developmental Biology, 2016
- Building machines that learn and think like peopleBehavioral and Brain Sciences, 2016
- Artificial General Intelligence: Concept, State of the Art, and Future ProspectsJournal of Artificial General Intelligence, 2014
- Ensemble Learning in Fixed Expansion Layer Networks for Mitigating Catastrophic ForgettingIEEE Transactions on Neural Networks and Learning Systems, 2013
- Machine Lifelong Learning: Challenges and Benefits for Artificial General Intelligence2011
- Incremental learning with partial instance memoryArtificial Intelligence, 2004
- Catastrophic forgetting in connectionist networksTrends in Cognitive Sciences, 1999
- Why there are complementary learning systems in the hippocampus and neocortex: Insights from the successes and failures of connectionist models of learning and memory.Psychological Review, 1995
- Catastrophic Forgetting, Rehearsal and PseudorehearsalConnection Science, 1995
- Catastrophic Interference in Connectionist Networks: The Sequential Learning Problem1989