ESDB: Expand the Shrinking Decision Boundary via One-to-Many Information Matching for Continual Learning With Small Memory
- 14 March 2024
- journal article
- research article
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Transactions on Circuits and Systems for Video Technology
- Vol. 34 (8) , 7328-7343
Abstract
Rehearsal methods based on knowledge distillation (KD) have been widely used in continual learning (CL). However, given memory constraints, few exemplars contain limited variations of previously learned tasks, impeding the effectiveness of KD in retaining long-term knowledge. The decision boundaries learned by the typical KD strategy overfit the limited exemplars, leading to “shrunk boundaries" of the old classes. To tackle this problem, we propose a novel KD strategy, called One-to-Many Information Matching method (O2MIM), which generates interpolated data by mixing samples between old and new classes, disentangles the supervision information from them and assigns supervision information to them in favor of the old classes. By doing so, the supervision information from a single exemplar can be matched with multiple information from different interpolated images. Moreover, O2MIM utilizes one trainable parameter to create an adaptive KD loss, thereby facilitating a flexible matching process with the designated supervision information. Consequently, O2MIM exploits the exemplar corset more effectively, expanding the shrunk decision boundaries towards the new classes. Next, to incorporate new classes into our classification model, we apply an effective classification training strategy to train a debiased classifier. Combining it with O2MIM, we propose the method of Expanding the Shrinking Decision Boundaries (ESDB), which simultaneously transfers knowledge from the old model via O2MIM and learns new classes by the classification training strategy. Extensive experiments demonstrate that ESDB achieves state-of-the-art performance on diverse CL benchmarks. We also confirm that O2MIM can be used with various label-mixing methods to improve overall performance in CL. The code is available at: https://github.com/CSTiger77/ESDB.Keywords
Funding Information
- STI 2030b??Major Project (2021ZD0200402)
- International Partnership Program of the Chinese Academy of Sciences (173211KYSB20200021)
- Beijing Natural Science Foundation (JQ23016)
- National Key Research and Development Program of China (2022YFB4500300)
- CAS Project for Young Scientists in Basic Research (YSBR-041)
This publication has 39 references indexed in Scilit:
- DualPrompt: Complementary Prompting for Rehearsal-Free Continual Learning2022
- Balanced softmax cross-entropy for incremental learning with and without memoryComputer Vision and Image Understanding, 2022
- FOSTER: Feature Boosting and Compression for Class-Incremental Learning2022
- Continual Learning with Dual Regularizations2021
- Knowledge Distillation: A SurveyInternational Journal of Computer Vision, 2021
- PODNet: Pooled Outputs Distillation for Small-Tasks Incremental Learning2020
- Memory-Efficient Incremental Learning Through Feature Adaptation2020
- Remix: Rebalanced Mixup2020
- Continual learning of context-dependent processing in neural networksNature Machine Intelligence, 2019
- Catastrophic Interference in Connectionist Networks: The Sequential Learning Problem1989