Continual Learning of Image Classes With Language Guidance From a Vision-Language Model
- 23 August 2024
- journal article
- research article
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Transactions on Circuits and Systems for Video Technology
- Vol. 34 (12) , 13152-13163
Abstract
Current deep learning models often catastrophically forget the knowledge of old classes when continually learning new ones. State-of-the-art approaches to continual learning of image classes often require retaining a small subset of old data to partly alleviate the catastrophic forgetting issue, and their performance would be degraded sharply when no old data can be stored due to privacy or safety concerns. In this study, inspired by human learning of visual knowledge with the effective help of language, we propose a novel continual learning framework based on a pre-trained vision-language model (VLM) without retaining any old data. Rich prior knowledge of each new image class is effectively encoded by the frozen text encoder of the VLM, which is then used to guide the learning of new image classes. The output space of the frozen text encoder is unchanged over the whole process of continual learning, through which image representations of different classes become comparable during model inference even when the image classes are learned at different times. Extensive empirical evaluations on multiple image classification datasets under various settings confirm the superior performance of our method over existing ones. The source code will be released publicly. The source code is available at https://github.com/Fatflower/CIL LG VLM/.Keywords
Funding Information
- National Natural Science Foundation of China (62071502)
- Major Key Project of Peng Cheng Laboratory (PCL2023A09)
- Guangdong Excellent Youth Team Program (2023B1515040025)
This publication has 35 references indexed in Scilit:
- Large language models in medicineNature Medicine, 2023
- Continual learning with Bayesian model based on a fixed pre-trained feature extractorVisual Intelligence, 2023
- Parameter-efficient fine-tuning of large-scale pre-trained language modelsNature Machine Intelligence, 2023
- Expert with Outlier Exposure for Continual Learning of New Diseases2022
- DualPrompt: Complementary Prompting for Rehearsal-Free Continual Learning2022
- Continual Learning of New Diseases with Dual Distillation and Ensemble Strategy2020
- iCaRL: Incremental Classifier and Representation Learning2017
- Overcoming catastrophic forgetting in neural networksProceedings of the National Academy of Sciences of the United States of America, 2017
- ImageNet: A large-scale hierarchical image database2009
- Catastrophic Interference in Connectionist Networks: The Sequential Learning Problem1989