CL3: Generalization of Contrastive Loss for Lifelong Learning
Open Access
- 23 November 2023
- journal article
- research article
- Published by MDPI AG in Journal of Imaging
- Vol. 9 (12) , 259
Abstract
Lifelong learning portrays learning gradually in nonstationary environments and emulates the process of human learning, which is efficient, robust, and able to learn new concepts incrementally from sequential experience. To equip neural networks with such a capability, one needs to overcome the problem of catastrophic forgetting, the phenomenon of forgetting past knowledge while learning new concepts. In this work, we propose a novel knowledge distillation algorithm that makes use of contrastive learning to help a neural network to preserve its past knowledge while learning from a series of tasks. Our proposed generalized form of contrastive distillation strategy tackles catastrophic forgetting of old knowledge, and minimizes semantic drift by maintaining a similar embedding space, as well as ensures compactness in feature distribution to accommodate novel tasks in a current model. Our comprehensive study shows that our method achieves improved performances in the challenging class-incremental, task-incremental, and domain-incremental learning for supervised scenarios.Keywords
Funding Information
- CSIRO’s Machine Learning and Artificial Intelligence Future Science Platform (MLAI FSP) (DP230101176)
- CSIRO’s Research Plus Postgraduate Scholarship (DP230101176)
- Australian Research Council (ARC) (DP230101176)
This publication has 37 references indexed in Scilit:
- Overcoming catastrophic forgetting in neural networksProceedings of the National Academy of Sciences, 2017
- Deep Residual Learning for Image Recognition2016
- ImageNet Large Scale Visual Recognition ChallengeInternational Journal of Computer Vision, 2015
- The stability-plasticity dilemma: investigating the continuum from catastrophic forgetting to age-limited learning effectsFrontiers in Psychology, 2013
- Neural dynamics of saccadic and smooth pursuit eye movement coordination during visual tracking of unpredictably moving targetsNeural Networks, 2012
- Gradient-based learning applied to document recognitionProceedings of the IEEE, 1998
- Catastrophic Forgetting, Rehearsal and PseudorehearsalConnection Science, 1995
- Catastrophic Interference in Connectionist Networks: The Sequential Learning Problem1989
- A massively parallel architecture for a self-organizing neural pattern recognition machineComputer Vision, Graphics, and Image Processing, 1987
- Studies of Mind and Brain1982