CKDF-V2: Effectively Alleviating Representation Shift for Continual Learning With Small Memory
- 2 June 2025
- journal article
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Transactions on Neural Networks and Learning Systems
- Vol. 36 (9) , 16171-16185
Abstract
In continual learning (CL), the newly arrived data are often out-of-distribution from the previous ones, causing drastic representation shift (RS) when updating the old model on the new data, leading to catastrophic forgetting. In this work, we propose feature boosting calibration (FBC) to tackle this problem. Specifically, an expanded module is trained to learn all the classes, including the old and new classes, discovering critical features missed by the original/old model. Then, an FBC network (FBCN) is trained to exploit these missed features to calibrate the old representations. As the missed features increase the information needed for distinguishing between the old and new classes, FBCN generates the calibrated ones with more transferable features, thus alleviating the RS. Next, given the limited memory to store samples of the old/learned classes, the data are severely imbalanced between the old and new classes. To cope with this problem, we propose blockwise knowledge distillation (BWKD), which splits the softmax layer into blocks according to class frequency and then distills each block separately, resolving data imbalance effectively. Building upon the two improvements, we propose a two-stage training framework for CL, named CKDF-V2, providing an enhanced version of the cascaded knowledge distillation framework (CKDF). Furthermore, we integrate it with a task-token expansion method to develop a novel approach for CL based on the vision transformer (ViT). Extensive experiments show that both a convolutional neural network (CNN) and ViT-based CKDF-V2 obtain favorable results across multiple CL benchmarks.Keywords
Funding Information
- Strategic Priority Research Program of Chinese Academy of Sciences (XDB1010302)
- CAS Project for Young Scientists in Basic Research (YSBR-041)
- International Partnership Program of CAS (173211KYSB20200021)
- Beijing Natural Science Foundation (JQ23016)
- Beijing Natural Science Foundation Haidian Original Innovation Joint Fund Project (L232035, L222154)
- National Natural Science Foundation of China (62476273, 62271452)
- Unveiling and Leading Projects of Xiamen (3502Z20241011)
- Major Science and Technology Plan Project on the Future Industry Fields of Xiamen City (3502Z20241027)
This publication has 33 references indexed in Scilit:
- AdaER: An adaptive experience replay approach for continual lifelong learningNeurocomputing, 2023
- Balanced softmax cross-entropy for incremental learning with and without memoryComputer Vision and Image Understanding, 2022
- Embracing Change: Continual Learning in Deep Neural NetworksTrends in Cognitive Sciences, 2020
- Feature Normalized Knowledge Distillation for Image Classification2020
- PODNet: Pooled Outputs Distillation for Small-Tasks Incremental Learning2020
- End-to-End Object Detection with Transformers2020
- Continual learning of context-dependent processing in neural networksNature Machine Intelligence, 2019
- iCaRL: Incremental Classifier and Representation Learning2017
- ImageNet: A large-scale hierarchical image database2009
- Catastrophic Interference in Connectionist Networks: The Sequential Learning Problem1989