Learning Without Memorizing
Authors:
Top Cited Papers
- 1 June 2019
- conference paper
- Published by Institute of Electrical and Electronics Engineers (IEEE)
- p. 5133-5141
Abstract
Incremental learning (IL) is an important task aimed at increasing the capability of a trained model, in terms of the number of classes recognizable by the model. The key problem in this task is the requirement of storing data (e.g. images) associated with existing classes, while teaching the classifier to learn new classes. However, this is impractical as it increases the memory requirement at every incremental step, which makes it impossible to implement IL algorithms on edge devices with limited memory. Hence, we propose a novel approach, called `Learning without Memorizing (LwM)', to preserve the information about existing (base) classes, without storing any of their data, while making the classifier progressively learn the new classes. In LwM, we present an information preserving penalty: Attention Distillation Loss (L_{AD}), and demonstrate that penalizing the changes in classifiers' attention maps helps to retain information of the base classes, as new classes are added. We show that adding L_{AD} to the distillation loss which is an existing information preserving loss consistently outperforms the state-of-the-art performance in the iILSVRC-small and iCIFAR-100 datasets in terms of the overall accuracy of base and incrementally learned classes.Keywords
All Related Versions
This publication has 10 references indexed in Scilit:
- Memory Aware Synapses: Learning What (not) to Forget2018
- Choose Your Neuron: Incorporating Domain Knowledge Through Neuron-Importance2018
- Tell Me Where to Look: Guided Attention Inference Network2018
- Less-Forgetful Learning for Domain Expansion in Deep Neural NetworksProceedings of the AAAI Conference on Artificial Intelligence, 2018
- Grad-CAM: Visual Explanations from Deep Networks via Gradient-Based Localization2017
- iCaRL: Incremental Classifier and Representation Learning2017
- Overcoming catastrophic forgetting in neural networksProceedings of the National Academy of Sciences, 2017
- Deep Residual Learning for Image Recognition2016
- Fully convolutional networks for semantic segmentation2015
- One-shot learning of object categoriesIEEE Transactions on Pattern Analysis and Machine Intelligence, 2006