Selective Freezing for Efficient Continual Learning
Authors:
- 2 October 2023
- conference paper
- Published by Institute of Electrical and Electronics Engineers (IEEE)
- p. 3542-3551
Abstract
This paper aims to tackle the challenges of continual learning, where sequential learning from a stream of tasks can lead to catastrophic forgetting. Simultaneously, it addresses the need to reduce the computational demands of large-scale deep learning models to mitigate their environmental impact. To achieve this twofold objective, we propose a method that combines selective layer freezing with fast adaptation in a continual learning context. We begin by conducting an extensive analysis of layer freezing in continual learning, revealing that certain configurations allow for freezing a substantial portion of the model without significant accuracy degradation. Leveraging this insight, we introduce a novel approach that optimizes plasticity on new tasks while preserving stability on previous tasks by dynamically identifying a subset of layers to freeze during training. Experimental results demonstrate the effectiveness of our approach in achieving competitive performance with manually-tuned freezing strategies. Moreover, we quantitatively estimate the reduction in computation and energy requirements achieved through our freezing strategy by considering the number of parameters and updates required for model training.Keywords
This publication has 23 references indexed in Scilit:
- Learning without ForgettingIEEE Transactions on Pattern Analysis and Machine Intelligence, 2017
- Mask R-CNN2017
- iCaRL: Incremental Classifier and Representation Learning2017
- What Learning Systems do Intelligent Agents Need? Complementary Learning Systems Theory UpdatedTrends in Cognitive Sciences, 2016
- Deep Residual Learning for Image Recognition2016
- Mastering the game of Go with deep neural networks and tree searchNature, 2016
- Carbon footprint: current methods of estimationEnvironmental Monitoring and Assessment, 2010
- Why there are complementary learning systems in the hippocampus and neocortex: Insights from the successes and failures of connectionist models of learning and memory.Psychological Review, 1995
- Connectionist models of recognition memory: Constraints imposed by learning and forgetting functions.Psychological Review, 1990
- Catastrophic Interference in Connectionist Networks: The Sequential Learning Problem1989