Co2L: Contrastive Continual Learning
Authors:
Top Cited Papers
- 1 October 2021
- conference paper
- Published by Institute of Electrical and Electronics Engineers (IEEE)
- p. 9496-9505
Abstract
Recent breakthroughs in self-supervised learning show that such algorithms learn visual representations that can be transferred better to unseen tasks than cross-entropy based methods which rely on task-specific supervision. In this paper, we found that the similar holds in the continual learning context: contrastively learned representations are more robust against the catastrophic forgetting than ones trained with the cross-entropy objective. Based on this novel observation, we propose a rehearsal-based continual learning algorithm that focuses on continually learning and maintaining transferable representations. More specifically, the proposed scheme (1) learns representations using the contrastive learning objective, and (2) preserves learned representations using a self-supervised distillation step. We conduct extensive experimental validations under popular benchmark image classification datasets, where our method sets the new state-of-the-art performance. Source code is available at https://github.com/chaht01/Co2L.Keywords
All Related Versions
Funding Information
- National Research Foundation of Korea
This publication has 12 references indexed in Scilit:
- Self-Supervised GANs via Auxiliary Rotation Loss2019
- PackNet: Adding Multiple Tasks to a Single Network by Iterative Pruning2018
- iCaRL: Incremental Classifier and Representation Learning2017
- Overcoming catastrophic forgetting in neural networksProceedings of the National Academy of Sciences, 2017
- ImageNet Large Scale Visual Recognition ChallengeInternational Journal of Computer Vision, 2015
- Dimensionality Reduction by Learning an Invariant Mapping2006
- SMOTE: Synthetic Minority Over-sampling TechniqueJournal of Artificial Intelligence Research, 2002
- Gradient-based learning applied to document recognitionProceedings of the IEEE, 1998
- Catastrophic Forgetting, Rehearsal and PseudorehearsalConnection Science, 1995
- Catastrophic Interference in Connectionist Networks: The Sequential Learning Problem1989