Ternary Feature Masks: zero-forgetting for task-incremental learning
Authors:
- 1 June 2021
- conference paper
- Published by Institute of Electrical and Electronics Engineers (IEEE)
- p. 3565-3574
Abstract
We propose an approach without any forgetting to continual learning for the task-aware regime, where at inference the task-label is known. By using ternary masks we can upgrade a model to new tasks, reusing knowledge from previous tasks while not forgetting anything about them. Using masks prevents both catastrophic forgetting and backward transfer. We argue – and show experimentally – that avoiding the former largely compensates for the lack of the latter, which is rarely observed in practice. In contrast to earlier works, our masks are applied to the features (activations) of each layer instead of the weights. This considerably reduces the number of mask parameters for each new task; with more than three orders of magnitude for most networks. The encoding of the ternary masks into two bits per feature creates very little overhead to the network, avoiding scalability issues. To allow already learned features to adapt to the current task without changing the behavior of these features for previous tasks, we introduce task-specific feature normalization. Extensive experiments on several finegrained datasets and ImageNet show that our method outperforms current state-of-the-art while reducing memory overhead in comparison to weight-based approaches.Keywords
All Related Versions
Funding Information
- Generalitat de Catalunya
This publication has 24 references indexed in Scilit:
- Overcoming catastrophic forgetting in neural networksProceedings of the National Academy of Sciences, 2017
- Deep Residual Learning for Image Recognition2016
- ImageNet Large Scale Visual Recognition ChallengeInternational Journal of Computer Vision, 2015
- Learning and Transferring Mid-level Image Representations Using Convolutional Neural Networks2014
- The stability-plasticity dilemma: investigating the continuum from catastrophic forgetting to age-limited learning effectsFrontiers in Psychology, 2013
- Human action recognition by learning bases of action attributes and parts2011
- Catastrophic forgetting in connectionist networksTrends in Cognitive Sciences, 1999
- Gradient-based learning applied to document recognitionProceedings of the IEEE, 1998
- Catastrophic Interference in Connectionist Networks: The Sequential Learning Problem1989
- A logical calculus of the ideas immanent in nervous activityBulletin of Mathematical Biology, 1943