Adaptive Deep Models for Incremental Learning
Authors:
Open Access
- 25 July 2019
- conference paper
- Published by Association for Computing Machinery (ACM)
- p. 74-82
Abstract
Recent years have witnessed growing interests in developing deep models for incremental learning. However, existing approaches often utilize the fixed structure and online backpropagation for deep model optimization, which is difficult to be implemented for incremental data scenarios. Indeed, for streaming data, there are two main challenges for building deep incremental models. First, there is a requirement to develop deep incremental models with Capacity Scalability. In other words, the entire training data are not available before learning the task. It is a challenge to make the deep model structure scaling with streaming data for flexible model evolution and faster convergence. Second, since the stream data distribution usually changes in nature (concept drift), there is a constraint for Capacity Sustainability. That is, how to update the model while preserving previous knowledge for overcoming the catastrophic forgetting. To this end, in this paper, we develop an incremental adaptive deep model (IADM) for dealing with the above two capacity challenges in real-world incremental data scenarios. Specifically, IADM provides an extra attention model for the hidden layers, which aims to learn deep models with adaptive depth from streaming data and enables capacity scalability. Also, we address capacity sustainability by exploiting the attention based fisher information matrix, which can prevent the forgetting in consequence. Finally, we conduct extensive experiments on real-world data and show that IADM outperforms the state-of-the-art methods with a substantial margin. Moreover, we show that IADM has better capacity scalability and sustainability in incremental learning scenarios.Keywords
Funding Information
- NSF (IIS-1814510)
- NSFC (61773198 61632004)
- National Key R$&$D Program of China (2018YFB1004300)
This publication has 18 references indexed in Scilit:
- A Survey on Ensemble Learning for Data Stream ClassificationACM Computing Surveys, 2017
- Mining Graphlet Counts in Online Social Networks2017
- A Bio-Inspired Incremental Learning Architecture for Applied Perceptual ProblemsCognitive Computation, 2016
- A survey on concept drift adaptationACM Computing Surveys, 2014
- Ensemble Learning in Fixed Expansion Layer Networks for Mitigating Catastrophic ForgettingIEEE Transactions on Neural Networks and Learning Systems, 2013
- Boosting for transfer learning2007
- Natural Gradient Works Efficiently in LearningNeural Computation, 1998
- Gradient-based learning applied to document recognitionProceedings of the IEEE, 1998
- A note on distributed estimation and sufficiencyIEEE Transactions on Information Theory, 1993
- Connectionist models of recognition memory: Constraints imposed by learning and forgetting functions.Psychological Review, 1990