SLCA: Slow Learner with Classifier Alignment for Continual Learning on a Pre-trained Model
Authors:
Top Cited Papers
- 1 October 2023
- conference paper
- Published by Institute of Electrical and Electronics Engineers (IEEE)
- p. 19091-19101
Abstract
The goal of continual learning is to improve the performance of recognition models in learning sequentially arrived data. Although most existing works are established on the premise of learning from scratch, growing efforts have been devoted to incorporating the benefits of pre-training. However, how to adaptively exploit the pre-trained knowledge for each incremental task while maintaining its generalizability remains an open question. In this work, we present an extensive analysis for continual learning on a pre-trained model (CLPM), and attribute the key challenge to a progressive overfitting problem. Observing that selectively reducing the learning rate can almost resolve this issue in the representation layer, we propose a simple but extremely effective approach named Slow Learner with Classifier Alignment (SLCA), which further improves the classification layer by modeling the class-wise distributions and aligning the classification layers in a post-hoc fashion. Across a variety of scenarios, our proposal provides substantial improvements for CLPM (e.g., up to 49.76%, 50.05%, 44.69% and 40.16% on Split CIFAR-100, Split ImageNet-R, Split CUB-200 and Split Cars-196, respectively), and thus outperforms state-of-the-art approaches by a large margin. Based on such a strong baseline, critical factors and promising directions are analyzed in-depth to facilitate subsequent research. Code has been made available at: https://github.com/GengDavid/SLCA.Keywords
All Related Versions
This publication has 29 references indexed in Scilit:
- DualPrompt: Complementary Prompting for Rehearsal-Free Continual Learning2022
- CoSCL: Cooperation of Small Continual Learners is Stronger Than a Big One2022
- REMIND Your Neural Network to Prevent Catastrophic Forgetting2020
- GDumb: A Simple Approach that Questions Our Progress in Continual Learning2020
- SpotTune: Transfer Learning Through Adaptive Fine-Tuning2019
- Large Scale Incremental Learning2019
- Memory Aware Synapses: Learning What (not) to Forget2018
- Overcoming catastrophic forgetting in neural networksProceedings of the National Academy of Sciences, 2017
- ImageNet: A large-scale hierarchical image database2009
- Why there are complementary learning systems in the hippocampus and neocortex: Insights from the successes and failures of connectionist models of learning and memory.Psychological Review, 1995